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Nine  software  cost  estimating  models  are  evaluated  to  determine  If  they 
satisfy  Air  Force  needs.  The  evaluation  considers  both  the  qualitative 
and  quantitative  aspects  cf  the  models'  outputs. 

Air  Force  needs  for  cost  estimates  are  established  by  the  Major  Weapon 
System  Acquisition  Process.  Associated  with  the  different  development 
phases  are  five  cost  estimating  situations.  Decisions  that  are  made - - 
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early  In  the  Acquisition  Process  require  software  cost  Information  that 
Includes  the  entire  life  cycle  for  complete  software  systems,  subse¬ 
quent  decisions  require  more  detailed  cost  information. 

Comparison  of  the  outputs  of  the  nine  test  models  with  the  requirements 
established  by  che  five  cost  estimating  situations  indicates  that  the 
models  are  able  to  satisfy  only  the  needs  of  the  earliest  phase  of  the 
Acquisition  Process.  The  models  perform  satisfactorily  for  the  purpose 
of  allocating  funds  for  software  acquisition,  but  they  fail  to  support 
such  needs  as  assessment  of  alternative  designs,  proposal  evaluation, 
or  project  management. 

Estimating  accuracy  w^^^easured  by  comparing  outputs  with  actual  ex¬ 
perience  using  data  from  three  organizations  representing  45  software 
development  projects.  The  best  model  performance  (Relative  root  mean 
square  error  i  40  percent)  is  obtained  when  a  model  is  calibrated  using 
representative  historical  data.  Calibration  was  found  to  have  greater 
effect  on  estimating  accuracy  than  the  model  form. 


Acc‘~- 

LVl  ■ 

U;.;  • 

• 

- 

1’; 

I  ! 

J '  • 

-  ' 

li  i L 

. .  ‘  — - -«i 

UNCLASSIFIED 


security  cl  ASSI^IC  ATIOH  Fn»#f#d) 


TABLE  OF  CONTENTS 


r 


SECTION  PAGE 


1 

INTRODUCTION  AND  SUMMARY 

1-1 

1.1 

COST  ESTIMATING  AND  SOFTWARE  COST  MODELS 

1-1 

1.2 

THE  AIR  FORCE  PERSPECTIVE  AND  SOFTWARE  COST 
MODEL  RELIABILITY 

1-1 

1.3 

OVERVIEW  OF  THE  SOFTWARE  MODEL  EVALUATION 

1-3 

1.4 

SUMMARY  OF  THE  REPORT 

1-5 

1.5 

SOME  DEFINITIONS 

1-10 

2 

MODEL  DESCRIPTIONS 

2-1 

3 

EVALUATION  CRITERIA 

3-1 

3.1 

INFORMATION  NEEDS 

3-2 

•3.2 

ACCURACY 

3-26 

3.3 

OTHER  EVALUATION  CRITERIA 

3-28 

4 

EVALUATION  PROCEDURE 

4-1 

4.1 

DEFINITIONS  OF  MODEL  AND  DATA  SET  VARIABLES 

4-1 

4.2 

MODEL  TYPES 

4-11 

4.3 

TEST  DATA  SETS 

4-18 

4.4 

MISSING  DATA 

4-21 

5 

RESULTS 

5-1 

5.1 

COMPLIANCE  WITH  AIR  FORCE  COST  INFORMATION  . 
NEEDS 

5-3 

5.2 

MODEL  ESTIMATING  ACCURACY 

5-21 

6 

ANALYSIS  OF  RESULTS 

6-1 

6.1 

ENVIRONMENT  i 

6-1 

6.2 

THE  EFFECTS  OF  INPUT  ESTIMATING  ERRORS 

6-2 

6.3 

MODEL  CALIBRATION 

6-6 

6.4 

THE  USE  OF  UNMEASURABLE  VARIABLES  AND 
PARAMETERS 

6-9 

6.5 

APPLICABILITY  OF  THE  EVALUATION 

6-11 

iii 


TABLE  OF  CONTENTS  (Cont) 


SECTION  _ 

7  RECOMMENDATIONS  7-1 

7.1  MODEL  DEVELOPMENT  7-1 

7.2  DATA  DEFINITION  AND  COLLECTION  7-3 


t 


iv 


Ca> 


TABLES 


NUMBER  _  page 

1  SUMMARY  OF  MODEL  CHARACTERISTICS  2-2 

2  SOFTWARE  LIFE  CYCLE  ACTIVITIEW  AND  PRODUCTS  3-11 

ESTIMATING  NEEDS  FOR  SOFTWARE  LIFE  CYCLE  PHASES  3-15 

4  FIVE  COST  ESTIMATING  SITUATIONS  3-18 

5  DECOMPOSITION  OF  SYSTEM  ELEMENTS  BY  MAJOR  WORK  3-23 

BREAKDOWN  STRUCTURE  DEFINITIONS 

6  SIZE  DEFINITIONS  USED  IN  THE  DIFFERENT  MODELS  4-8 

7  SUMMARY  OF  MODEL  COMPLIANCE  WITH  AIR  FORCE  ESTIMATING  5-17 

REQUIREMENTS 

8  SUMMARY  OF  MODEL  ESTIMATING  PERFORMANCE  5-22 

9  EFFECTS  OF  ENVIRONMENT  AND  MODEL  TYPE  OF  ESTIMATING  5-24 

PERFORMANCE 

10  PAIRWISE  COMPARISONS  OF  ESTIMATING  PERFORMANCE  5-26 

n  AVERAGE  ESTIMATING  PERFORMANCE  5-29 

12  INPUTS  FOR  MODFu  F,  BOEING  COMPUTER  SERVICES  6-4 

13  INPUTS  FOR  MODEL  G,  MICRO  ESTIMATING  PROCEDURE  6-5 

A-1  -UGGESTED  UTILIZATION  OF  ESTIMATING  RELATIONSHIPS  A-20 

OEVELOPi'ENT  MANPOWER 

A-2  SOFTWARE  DEVELOPMENT  MANPOWER  ESTIMATING  ALGORITHMS  A-21 

REFLECTING  DEVELOPMENT  ENVIRONMENT 

A-3  SOFTWARE  PROGRAM  COSTS  RIPPLE  EFFECT  A-38 

A-4  MIX  CATEGORIES  A-53 

A-5  TYPICAL  PLTFM  VALUES  A-57 

A-6  TYPICAL  CPLX  VALUES  A-58 

A-7  SUMMARY  OF  PROVISIONAL  SOFTWARE  ESTIMATING  A-83 

RELATIONSHIPS  (SEE  NOTE  A) 

A-8  ACTIVITIES  AS  A  FUNCTION  OF  SOFTWARE  DEVELOPMENT  PHASE  A-87 

A-9  COST  MATRIX  DATA,  SHOWING  ALLOCATION  OF  RESOURCES  A-88 

AS  A  FUNCTION  OF  ACTIVITY  BY  PHASE 

C-1  MODEL  ESTIMATING  PERFORMANCE  -  AEROSPACE  CORPORATION.  C-2 

COMMERCIAL 

C-2  MODEL  ESTIMATING  PERFORMANCE  -  AEROSPACE  CORP,  C-3 

DSDC 

C-3  MODEL  ESTIMATING  PERFORMANCE  -  AEROSPACE  CORPORATION,  C-4 

SEL 


V 


TABLES  (Cont) 


NUMBER 

PAGE 

C-4 

MOOEL  ESTIMATING  PERFORMANCE  - 
DSDC 

BOEING  COMPUTER  SERVICES 

C-5 

C-5 

MODEL  ESTIMATING  PERFORMANCE  - 
DSDC 

DOD  MICRO  PROCEDURE. 

C-c 

C-6 

MODEL  ESTIMATING  PERFORMANCE  - 
DSDC 

DOTY  ASSOCIATES,  INC., 

C-7 

C-7 

MODEL  ESTIMATING  PERFORMANCE  - 

FARR  AND  ZAGORSKI,  DSDC 

C-8 

C-8 

MOOEL  ESTIMATING  PERFORMANCE  - 

PRICE  S,  COMMERCIAL 

C-9 

C-9 

MODEL  ESTIMATING  PERFORMANCE  - 

PRICE  S.  DSDC 

C-10 

-  O 

MOOEL  ESTIMATING  PERFORMANCE  - 

PRICE  S,  SEL 

C-1  i 

■ ' 

MODEL  ESTIMATING  PERFORMANCE  - 

SLIM,  COMMERCIAL 

C-1I. 

C-i2 

MODEL  ESTIMATING  PERFORMANCE  - 

SLIM,  DSDC 

C-13 

C-13 

MOOEL  ESTIMATING  PERFORMANCE  - 

SLIM,  SEL 

C-U 

C-14 

MOOEL  ESTIMATING  PERFORMANCE  - 

TECOLOTE,  DSDC 

C-15 

C-15 

MOOEL  ESTIMATING  PERFORMANCE  - 

WOLVERTON,  DSDC 

C-16 

C-16 

MODEL  ESTIMATING  PERFORMANCE  - 
EQUATION 

RECALIBRATED  SIZE 

C-17 

C-17 

SUMMARY  OF  MODEL  ESTIMATING  PERFORMANCE 

C-18 

0-1 

AEROSPACE  AND  TECOLOTE  MODELS 

D-2 

0-2 

BOEING  COMPUTER  SERVICES 

D-3 

0-3 

MICRO  ESTIMATING  PROCEDURE 

D-4 

0-4 

DOTY 

D-5 

0-5 

FARR  &  ZAGORSKI  MODEL 

0-6 

0-6 

PRICE  S 

D-7 

0-7 

PRICE  S 

0-8 

0-8 

PRICE  S 

D-9 

0-9 

SLIM 

D-10 

0-10 

WOLVERTON  MODEL 

D-11 

0-11 

RECALIBRATED  SIZE  EQUATION 

0-12 

VI 


FIGURES 


NUMBER  _ _  PAGE 


1  Major  Weapon  System  Life  Cycle  3-5 

2  The  Software  Life  Cycle  3-9 

3  Relationship  Between  Weapon  System  and  3-10 

Software  Life  Cycles 

4  The  Definition  of  the  System  Elements  and  3-24 

Their  Relationship  to  the  Software  Life 

Cycle  and  WBS 

5  Problems  in  Compatibility  Between  Data  Sets  4-22 

and  Model  Variables 

6  Comparison  Between  Estimating  Requirements  5-5 

and  Model  Outputs  -  Aerospace  Corporation 

7  Comparison  Between  Estimating  Requirements  and  5-6 

Model  Outputs  -  Boeing  Computer  Service 

8  Comparison  Betv^een  Estimating  Requirements  and  5-S 

Model  Outputs  -  DoD  Micro-Procedure 

9  Comparison  Between  Estimating  Requirements  and  5-9 

Model  Outputs  -  Doty  Associates 

10  Comparison  Between  Estimating  Requirements  and  5-10 

Model  Outputs  -  Farr  &  Zagorski 

11  Comparison  Between  Estimating  Requirements  and  5-11 

Model  Outputs  -  PRICE  S 

12  Comparison  Between  Estimating  Requirements  and  5-13 

Model  Outputs  -  SLIM 

13  Comparison  Between  Estimating  Requirements  and  5-14 

Model  Outputs  -  Tecolote 

14  Comparison  Between  Estimating  Requirements  and  5-15 

Model  Outputs  -  Wolverton 


Allocation  of  Work  Breakdown  Structure  Elements 
to  Life  Cycle  Phases 


vi  i 


15 


5-19 


FIGURES  (Con't.) 


NUMBER 

PAGE 

A-1 

Sequence  of  Calculations  in  PRICE  S 

A- 37 

A- 2 

Standard  PRICE  S  Cost  Report 

A-45 

A- 3 

Sensitivity  Analyses 

A-46 

A-4 

PRICE  S  Inputs 

A-49 

A- 5 

Computation  of  LEVEL 

A-51 

A-6 

Effect  of  UTIL  on  COST 

A-56 

A- 7 

Cost  per  Object  Instruction  Versus 

Relative  Degree  of  Difficulty 

A- 86 

vi  i  i 


1 


INTRODUCTION  AND  SUMMARY 


1.1  COST  ESTIMATING  AND  SOFTWARE  COST  MODELS 

Cost  estimating  is  an  integral  part  of  the  Air  Force  major  weapon 
system  acquisition  process  [1]  [2]  [3].  The  Air  Force  manages  the  weapon 
system  life  cycle  by  continually  balancing  performance,  cost,  and  risk  for 
the  system  and  its  components.  Throughout  the  weapon  system  life  cycle  it 
is  necessary  to  estimate  the  cost  of  part  or  all  of  the  system  over  a  part 
or  all  of  its  development  and  operational  life. 

Computers  are  an  increasingly  important  part  of  Air  Force  weapon 
systems  in  terms  of  both  function  and  cost  [4]  [5].  Until  recently,  most 
of  the  cost  analysis  and  planning  related  to  computer  subsystems  was  directed 
to  the  hardware.  However,  increased  capabilities  and  reductions  in  the  cost 
of  hardware  have  had  the  effect  of  increasing  the  amount  of  software  needed 
for  each  system  and  its  cost  relative  to  the  cost  of  the  hardware.  It  is  now 
often  necessary  to  budget  large  portions  of  the  system  life  cycle  cost  to  the 
development  and  maintenance  of  these  software  components  [6j  [7]  [3]. 
Therefore,  more  attention  is  being  given  to  the  methods  used  for  making 
estimates  of  the  resources  to  be  invested  in  the  software  subsystems. 

A  software  cost  model  is  a  systematic  procedure  that  relates  cost 
to  certain  variables  or  cost  factors.  A  number  of  such  models  are  available 
to  cost  analysts.  The  Air  Force  has  commissioned  this  study  to  examine  some 
of  these  models  to  learn  the  extent  to  which  they  satisfy  Air  Force  needs  and 
to  learn  how  the  quality  of  software  estimating  can  be  improved. 

1.2  THE  AIR  FORCE  PERSPECTIVE  AND  SOFTWARE  COST  MODEL  RELIABILITY 

There  are  cost  estimating  situations  in  which  the  Air  Force  must 

consider  the  effect  on  software  cost  of  who  builds  it  or  how  it  is  built. 
Tiierefore,  it  is  useful  to  divide  cost  factors  into  those  that  describe  the 
product  under  development  and  those  that  describe  the  manner  in  which  it  is 
built.  Cost  factors  other  than  those  that  describe  the  product  are  affected 
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by  the  selection  of  a  development  organization  or  the  development  process. 
These  non-product  cost  factors  are  difficult  to  identify  and  measure.  In  the 
case  of  hardware  porducts  they  include  such  things  as  experience,  tools,  and 
facilities.  Given  the  proper  adaptation  of  definitions,  the  same  terms  are 
applicable  to  software  development.  In  either  case,  these  environmental 
factors  may  appear  explicitly  in  cost  estimating  procedures  or,  more  often, 
they  may  influence  the  applicability  of  a  given  model  to  a  given  development 
environment  in  some  unknown  way.  A  major  consideration  in  evaluating  models 
for  Air  Force  use  is  measuring  the  ability  of  the  model  to  define  the  environ¬ 
mental  parameters.  This  is  because  the  Air  Force  must  always  make  its 
estimates  at  arms  length.  It  must  know  how  the  cost  of  software  is  influenced 
oy  how  it  is  developed  and  who  develops  it. 

It  may  be  helpful  to  compare  methods  for  estimating  software  cost 
with  those  used  for  estimating  computer  hardware  cost.  Computer  hardware 
cost  estimating  is  more  advanced  than  software  cost  estimating.  This  is 
because  there  has  been  a  recognized  need  for  it  for  a  longer  time  and 
because  cost  estimating  techniques  that  were  developed  for  other  electronic 
components  were  adaptable  to  computers.  Hardware  possesses  readily 
identifiable  measures  of  size  and  performance  that  have  been  correlated  with 
cost  [9]  [10],  Given  a  hardware  product  with  specified  physical  and 
functional  characteristics ,  methods  exist  [11]  [12]  [13]  [14]  for  considering 
the  effects  on  cost  of  non-product  factors  such  as  state  of  the  art  advance, 
experience,  learning  and  manufacturing  techniques.  Therefore,  it  is  possible 
to  make  early  cost  estimates  using  average  industry  performance  (or  some 
desired  increase  over  the  existing  average);  and  then,  in  later  phases  of 
the  life  cycle,  it  is  possible  to  evaluate  proposals  and  give  proper  credit 
for  new  approaches  and  to  identify  high  risk  or  infeasible  concepts. 

Although  software  costs  are  also  affected  by  non-product  factors 
[15]  [15]  [17],  there  are  no  reliable  procedures  for  quantitatively 
describing  their  effects  on  cost.  The  most  common  existing  procedure 
for  accounting  for  differences  in  development  methods  or  organizational 
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experience  is  to  base  model  estimates  on  historical  experience  similar 
to  the  proposed  development  environment.  However,  there  is  very  little 
objective  basis  for  distinguishing  among  projects  to  determine  whether 
they  are  truly  applicable  to  the  proposed  environment.  This  capability  is 
essential  if  the  Air  Force  is  to  properly  evaluate  software  development 
and  maintenance  proposals  from  different  organizations. 

There  are  several  reasons  why  software  cost  estimates  are  not  as 
reliable  as  those  for  hardware  [18]  [19]: 

•  Software  development  engineering  is  a  relatively  new  discipline. 

•  Software  design  and  development  methods  have  been  affected  by 
the  explosive  development  of  computer  hardware  which  has  changed 
the  cost  incentives  relating  software  and  hardware. 

•  Software  has  only  recently  become  a  major  cost  item  in  the  weapon 
system  life  cycle. 

•  The  relationships  between  cost  and  generally  accepted  cost  factors 
are  not  established. 

•  Reliable  historical  data  on  software  costs  are  almost  nonexistant. 

None  of  these  deterrents  to  reliable  software  cost  estimates  represents 
an  insurmountable  barrier.  One  purpose  of  this  project  is  to  evaluate  a 
number  of  existing  cost  estimating  techniques  or  models  to  learn  how  to 
overcome  past  problems. 

1.3  OVERVIEW  OF  THE  SOFTWARE  MODEL  EVALUATION 

The  evaluation  design  stems  from  the  belief  that  any  evaluation 
of  the  merits  of  different  approaches  to  a  given  objective  (i.e.,  obtaining 
good  cost  estimates)  should  be  based  on  the  comparison  of  the  approaches 
with  some  standard.  To  permit  the  evaluation  to  be  only  a  comparison 
of  how  the  several  existing  software  models  are  alike  and  different  Is 
an  abdication  of  the  evaluator's  prerogative  to  impose  the  standard  of 
measurement.  To  look  at  all  existing  models,  make  a  list  of  their 
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characteristics  and  then  show  how  each  compares  with  all  the  others, 
makes  the  assumption  that  the  Air  Force  needs  are  represented  in  the  study 
population.  It  implies  that  there  are  no  requirements  other  than  those 
that  prompted  the  designs  of  the  test  subjects.  Furthermore,  it  fails 
to  consider  whether  the  existing  models  have  satisfied  even  their  creators' 
objectives . 

A  detailed  statement  of  Air  Force  estimating  needs  (Section  2.1), 
establishes  objective  standards  for  cost  models  that  avoids  features  or 
qualities  of  existing  models  that  may  be  expensive  or  difficult  to  achieve, 
and  which  are  not  needed.  It  is  then  relatively  easy  to  compare  model 
characteristics  and  evaluation  objectives.  Since  the  evaluation  is  based 
on  satisfaction  of  needs,  this  approach  provides  a  ready  basis  for 
establishing  priorities  for  possible  research  programs. 

Past  comparative  studies  of  software  cost  models  [20]  [21]  [22] 

[23]  [24]  [25]  [26]  [27],  have  provided  descriptions  of  model  features  and 
discussed  different  methods  for  making  estimates.  Several  studies  [28] 

[29]  [30]  have  been  published  describing  estimating  experience  with  the  PRICE  S 
model.  But  there  has  been  no  comprehensive  analysis  of  predictions 
relative  to  needs  nor  a  comparative  analysis  of  estimating  performance 
using  data  from  different  environments.  This  evaluation  compares 
estimating  performance  using  three  different  development  data  sets. 

This  is  an  important  part  of  the  evaluation  design  because  several 
reports  indicate  that  environment  is  a  significant  factor  affecting  model 
estimating  accuracy  [31]  [32]  [33].  The  use  of  three  data  sets  is 
intended  to  help  identify  model  features  that  are  sensitive  to  environ¬ 
mental  change.  Controlling  these  factors  should  help  uncover  other 
determinants  of  accuracy. 

If  the  objective  of  the  accuracy  evaluation  was  to  determine  which 
of  the  nine  models  is  the  most  accurate  estimator  on  a  given  data  set,  it 
would  only  be  necessary  to  execute  the  models  using  the  same  data  and 


tabulate  the  difference  between  the  predicted  and  measured  values  of  the 
test  variable.  Such  an  evaluation,  however,  would  not  tell  the  Air  Force 
whether  the  measured  accuracy  would  be  obtained  for  all  estimating 
situations  or  guide  future  model  development  by  indicating  model  attributes 
that  contribute  to  higher  estimating  accuracy. 

The  evaluation  of  model  accuracy  should  address  the  following 
considerations : 

•  The  effect  of  the  software  development  environment  on  model 
performance. 

•  Attributes  of  the  environment  that  are  associated  with  the  best 
and  worst  performance  of  a  model.  That  is,  factors  that  indicate 
when  it  is  best  to  use  a  given  model  and  when  it  should  not  be  used. 

•  The  effect  on  the  accuracy  measurement  of  incomplete  input  sets 
among  the  test  data. 

•  The  characterization  of  model  structures  in  a  way  that  will  help 
to  identify  correlations  between  structural  attributes  and 
estimating  performance. 

1.4  SUMMARY  OF  THE  REPORT 

The  material  in  this  report  is  presented  in  much  the  same  sequence 
that  the  evaluation  project  was  completed.  The  models  to  be  evaluated 
were  selected  and  analyzed,  the  evaluation  criteria  including  Air  Force 
cost  estimating  needs  and  accuracy  were  established,  data  sets  were 
identified  and  qualified,  and  finally  the  evaluation  protocol  was  executed 
and  the  results  analyzed.  Specifically,  the  pertinent  sections  of  the 
report  are: 

2  Descriptions  of  the  Evaluated  Models 

3  Definition  of  the  Evaluation  Procedure 

4  The  Establishment  of  the  Evaluation  Criteria 

5  Execution  of  the  Evaluation  Procedure 

6  Analysis  of  the  Results  of  the  Evaluation 

7  Recommendations  for  Future  Model  Development 
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Section  2  presents  the  general  selection  criteria  used  for  the 
models  and  includes  a  one-page  summary  of  each  model.  The  models  are 
described  according  to  the  three  structural  types  developed  in  Section  4.2, 
their  method  of  making  their  initial  and  subsequent  estimates,  and  their 
outputs. 

Section  3  axplains  the  evaluation  criteria  established  for  Air  Force 
cost  estimating  needs  and  the  measurement  of  prediction  accuracy.  The 
cost  estimating  information  needs  are  established  by  the  Major  Weapon 
System  Acquisition  Process  (Section  3.1).  Consistent  with  this  process 
is  the  Air  Force  Software  Life  Cycle  and  a  comprehensive  Work  Breakdown 
Structure  (Appendix  B).  The  Weapon  System  Acquisition  Process  gives  rise 
to  five  cost  estimating  situations  that  should  be  supported  by  cost 
models.  The  Software  Life  Cycle  defines  the  set  of  activities  and  events 
that  describe  the  boundaries  of  the  cost  estimates.  The  Work  Breakdown 
Structure  establishes  the  elements  of  the  product  within  the  life  cycle 
phases  that  must  be  identifiable  by  separate  cost  values.  The  evaluation 
of  the  extent  to  which  existing  models  satisfy  the  five  estimating  situa¬ 
tions  is  made  by  comparing  the  model  outputs  with  the  requirements  in 
terms  of  scope  and  detail . 

Estimating  accuracy  may  be  measured  using  different  variables. 
Section  3.2  discusses  several  alternative  methods  and  explains  why  the 
Average  Relative  Root  Mean  Square  Error  was  selected. 

A  large  part  of  the  effort  spent  on  the  project  was  devoted  to 
obtaining  accurate  descriptions  of  model  inputs  and  outputs  (Section  3.1). 
Most  published  model  descriptions  are  vague  in  their  definitions  of  their 
variables.  It  is  difficult  to  know  exactly  which  cost  elements  are 
included  in  the  model  estimates.  One  common  problem  was  the  variations 
in  the  use  of  the  most  frequently  used  input:  size  of  code.  Many 
different  definitions  were  encountered. 
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Section  4.2  describes  the  three  categories  used  to  designate  the 
model  structures: 

•  Regression 

•  Heuristic 

•  Phenomenological 

Section  4.3  describes  the  three  organizations  that  contributed  data 
to  the  evaluation  and  some  of  the  processes  used  to  obtain  and  qualify  it. 

The  nine  test  models  are  associated  with  such  a  large  number  of 
different  input  and  output  variables  that  none  of  the  data  sets  was  rich 
enough  to  provide  measured  values  of  each.  Section  4.4  describes  how  the 
missing  data  items  were  handled. 

The  results  of  the  evaluation  are  presented  in  Section  5.  Section  5.1 
describes  how  well  the  models  satisfy  the  cost  information  needs  established 
by  the  five  cost  estikjating  situations,  the  Software  Life  Cycle  definitions 
and  the  Work  8reakdown\tructure.  'Section  5.2  contains  the  results  of  the 
accuracy  measurements.  Estimating  performance  is  related  to  model  and 
environmental  characteristics. 

The  evaluation  ihdicates  that  the  performance  of  the  models  tested 
is  very  sensitive  to  the  development  environment.  Within  an  environment 
characterized  by  similar  projects,  personnel  experience  and  management 
techniques,  the  most  accurate  models  achieved  an  average  estimating  error 
of  about  25  percent  on  the  basis  of  the  root  mean  square  error.  However, 
a  model  that  exhibits  such  performance  on  one  data  set  may  demonstrate  an 
average  error  approaching  100  percent  on  another.  Even  within  a  single 
environment  one  of  the  best  performing  models  has  an  error  range  of  +  50 
percent.  These  error  measurements  were  made  after  the  models  were  calibrated 
on  the  test  data  sets.  Therefore,  the  accuracy  is  greater  than  wouiq  ne 
expected  when  estimating  a  new  project. 


These  results  indicate  that  in  virtually  all  estimating  situations 
there  are  factors  that  are  not  properly  accounted  for  by  the  models  tested. 
These  factors  are  affected  by  changes  occurring  between  environments  and 
within  an  environment. 

The  results  of  the  evaluation  are  sumriarized  as  follows: 

A  comparison  of  the  outputs  of  the  models  under  investigation  with 
the  Air  Force  estimating  needs  indicates  that; 

•  The  supporting  materials  for  most  of  the  models  do  not  clearly 
state  the  elements  included  in  their  estimates  and  are  not  precise 
about  their  definitions. 

•  The  existing  models  are  better  able  to  satisfy  information  needs 
early  in  the  acquisition  life  cycle. 

•  None  of  the  models  included  in  this  study  fully  satisfy  the  Air 
Force  need  for  information  eitner  with  regard  to  scope  or  detail, 

•  The  models  tend  to  be  phase  oriented  and  do  not  properly  describe 
activities  that  cross  phase  boundaries.  This  precludes  obtaining 
data  compatible  with  both  management  planning  (phase  related)  and 
product  cost  (WBS). 

•  Although  most  of  the  models  use  the  summation  of  program  or  module 
sizes  to  make  their  cost  estimate,  only  one  model  studied  provides 
for  keeping  track  of  the  cost  on  a  component  basis  and  accounts 
for  the  cost  of  system  integration.  None  of  the  models  provide 
for  all  four  levels  of  system  definition  called  for  in  the  Work 
Breakdown  Structure  (Ref.  Appendix  B). 

Based  on  the  relative  root  mean  square  error  measure  of  performance: 

•  Recalibration*  is  the  primary  factor  contributing  to  the 
differences  in  estimating  performance  among  the  models  tested. 

•  The  contribution  of  model  structure*  to  estimating  accuracy  is 
not  significant  when  the  models  have  been  calibrated  to  the 
development  environment*. 

Definitions  of  these  terms  are  given  in  Section  l.b. 


•  The  development  environment  significantly  affects  the  relative 
performance  of  the  models  tested. 

•  The  effect  of  development  environment  on  estimating  performance 
precludes  the  possibility  of  obtaining  generally  applicable 
measures  of  model  performance  without  applying  additional  controls. 

•  Models  that  do  not  use  size  as  an  input  may  perform  as  well  as 
those  that  do. 

•  The  average  RMS  Error  for  all  tested  models  is  unacceptably  large 
for  Air  Force  estimating  purposes. 

•  The  best  performance  obtained  by  any  group  of  the  models  tested  is 
not  adequate  for  Air  Force  needs. 

Caution  must  be  exercised  to  avoid  extending  the  interpretation  of 
the  results  of  the  accuracy  measurements  beyond  the  constraints  of  this 
study.  Section  6  discusses  five  considerations  affecting  the  reliability 
of  the  measurements. 

Section  6.1  explains  how  the  development  environment  affects 
estimating  performance  and  the  rankings  of  the  models. 

Section  6.2  considers  the  effects  on  the  accuracy  measurement  of 
errors  in  the  estimated  input  values. 

Section  6.3  describes  the  methods  used  to  calibrate  the  models  on 
the  historical  data  sets  and  the  implication  for  the  evaluation. 

Section  6.4  explains  the  use  by  some  models  of  parameters  and 
variables  that  can  never  be  measured. 

The  recommendations  for  future  model  development  are  divided  into 
two  parts.  Section  7.1  describes  needs  for  new  experiments  identified  during 
this  project.  Section  7.2  makes  recommendations  for  better  data  definition 
and  collection. 


1-9 


1.5  SOME  DEFINITIONS 


The  discussions  in  this  document  include  several  terms  that  have 
specific  meanings  within  the  context  of  the  evaluation.  They  are  defined 
here  to  clarify  the  presentation  of  the  results. 

Model  Structure.  A  cost  estimating  model  is  considered  to  be  the  specific 
representation  of  the  model  structure  and  its  associated  parameters  that 
is  to  be  executed  in  a  given  cost  estimating  situation.  A  model  structure 
includes  imputs,  a  calculation  process  and  outputs.  It  is  the  formal 
representation  of  how  the  outputs  are  related  to  the  cost  driving  variables 
or  inputs.  In  addition  to  the  inputs,  which  represent  the  attributes  of  a 
specific  project  or  development  effort,  there  are  parameters  of  constants 
that  complete  the  quantification  of  the  model.  The  parameters  may  be  obtained 
empirically  from  representative  past  projects  or  they  may  be  subjective. 

They  determine  and  represent  the  universe  of  environments  for  which  the  model 
is  applicable.  In  some  cases,  different  parameters  are  given  for  different 
estimating  situations  (e.g.  Doty);  in  others,  the  models  are  presented  with¬ 
out  restrictions  on  the  applicability  of  the  parameters.  Two  models  (PRICE  S 
and  SLIM)  identify  the  parameters  and  provide  means  for  estimating  them  for 
any  environment. 

Throughout  this  report  the  term  "model"  refers  to  the  combination 
of  the  "model  structure"  and  values  of  the  parameters.  The  "model  structure" 
is  the  representation  of  the  estimating  hypothesis.  Our  ultimate  objective 
is  to  relate  the  attributes  of  the  model  structure  to  accuracy. 

Cal i bration .  The  process  by  which  values  of  model  parameters  are  obtained 
for  a  given  cost  estimating  situation  is  called  "calibration".  The  calib¬ 
ration  of  a  model  structure  may  be  performed  using  formal  curve  fitting 
methods  on  a  representative  historical  data  set,  by  using  an  execution 
mode  of  the  model,  or  by  selecting  values  from  experience.  An  important 
consideration  in  this  evaluation  was  the  proper  selection  of  representative 
data  and  methods  for  calibrating  the  model  structures. 
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Environment.  This  is  a  general  term  used  to  describe  the  source  of 
influencing  forces  that  are  external  to  the  product  being  developed.  As 
was  mentioned  before,  it  is  conceptually  helpful  when  analyzing  model 
structures  to  divide  the  cost-driving  factors  into  two  groups:  factors 
that  describe  the  product  and  are  therefore  unchanged  by  how  or  where 
the  development  is  completed;  and  factors  that  affect  the  resources  needed 
to  develop  the  product  but  are  independent  of  its  characteristics.  The 
first  group  are  usually  referred  to  as  input  variables  and  the  second 
group  constitutes  the  environmental  parameters.  Examples  of  environmental 
factors  are:  type  of  development  organization,  type  of  contract,  method 
of  project  organization,  development  methods,  supporting  software, 
facilities,  and  description  and  availability  of  computer  hardware. 
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2  MODEL  DESCRIPTIONS 

Software  cost  estimating  models  were  selected  for  evaluation  for 
one  or  more  of  the  following  reasons: 

•  Possessing  a  unique  structure 

•  Representing  a  common  type  of  structure 

•  A  representative  choice  of  input  variables 

•  A  unique  choice  of  input  variables 

•  Widespread  use 

•  Otherwise  interesting  to  the  Air  Force. 

The  following  models  were  evaluated: 

•  Aerospace  Corporation 

•  Boeing  Computer  Services 

•  DoD  Micro  Estimating  Procedure 

•  Doty  Associates,  Inc. 

•  Farr  and  Zagorski 

•  PRICE  S 

•  SLIM 

•  Tecolote  Research  Corporation 

•  Wolverton 

Detailed  descriptions  of  the  models  including  their  inputs  and 
outputs  are  prese”ted  in  Appendix  A.  The  following  are  one-page  summaries 
of  the  models  (Table  1)  ‘hat  describe  the  characteristics  upon  which  inferences 
concerning  the  contribution  of  model  structure  to  performance  are  based. 

These  attributes  include: 

•  Model  type 

•  Estimating  Procedure 

-  Level  of  initial  estimate 

-  Method  of  making  initial  estimate 

-  Method  of  making  subsequent  estimates 

•  Characterization  of  productivity 

•  Outputs 


AEROSPACE  CORPORATION 


STRUCTURE 

Type.  Regression 

First  estimate.  Development  effort. 

Single  parameter 

Subsequent  estimates.  No  further  breakdown  of  effort. 

Development  effort  is  calculated  given  the  number  of  instructions  using 
an  estimating  equation  of  the  form: 

MM  =  al^ 

where  MM  =  Manmonths  of  development  effort 
I  *  Number  of  instructions 
a,b  =  Constants 

OUTPUTS 

Effort. 

Scope.  Assumed  to  be  Analysis  through  System  Test. 
Detail.  System  or  CPCI  level. 


Table  1  Summary  of  Model  Characteristics 
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BOEING  COMPUTER  SERVICES 


STRUCTURE 


Type.  Heuristic 

First  estimate.  Development  effort. 

Multi -parameter 

Subsequent  estimates.  Allocations  using  fixed  ratios  followed  by  phase- 

related  adjustments. 

The  system  is  divided  into  five  types  of  software  and  the  number  of  delivered 
instructions  is  estimated  for  each  component.  The  system  development  effort 
is  obtained  by  multiplying  the  productivity  rate  in  manmonths  per  instruction 
for  each  type  of  software  and  adding  the  values  for  the  components.  The 
development  effort  is  divided  into  six  life  cycle  phases  using  fixed  ratios. 
The  phase  estimates  are  adjusted  for  certain  development  and  software  charac¬ 
teristics  and  recombined  to  form  a  revised  total  development  effort. 


OUTPUTS 


Effort. 

Scope.  Analysis  through  System  Test 
Detail.  System  level 


Table  1  'Cont)  Sumnary  of  Model  Characteristics 
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POD  MICRO  PROCEDURE 


STRUCTURE 


Type.  Heuristic 

First  estimate.  Portion  of  development  effort  (Direct 
development  effort) 

Mul ti -parameter 

Subsequent  estimates.  ,  Fixed  ratios 


Net  development  effort  is  calculated  using  an  estimating  equation 
that  includes  software  function  and  complexity  variables  along  with 
experience  measures. 


A  constant  factor  is  used  to  estimate  gross  development  effort 
which  then  divided  into  phases  using  ratios. 

OUTPUTS 

Effort. 

Scope.  Analysis  through  Installation 
Detail.  System  level 


Table  1  '.Co, it)  Summary  of  Model  Characteristics 
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DOTY 


STRUCTURE 


Type.  Regression 


First  estimate.  Development  effort. 

Mul ti -parameter 


Effort  is  related  to  size  and  type  of  code  by  esp' mating  equations. 
For  small  systems  the  effects  of  14  environmental  parameters  are 
included  using  a  product  function. 


OUTPUTS 

Effort. 

Scope.  Detailed  Design  through  Coding  and  Checkout 
Detail.  Total  effort  for  a  CPCI 
Development  time. 


Table  1  (Cont)  Sl  .nary  of  Model  Characteristics 
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FARR  AND  ZAGORSKI 


STRUCTURE 

Type.  Regression 

First  estimate.  Development  effort. 

Multi -parameter 

Subsequent  estimates.  No  further  breakdown  of  effort 

Effort  is  related  to  5  predictor  variables  by  an  estimating  equation. 

OUTPUTS 

Effort. 

Scope.  Detailed  design  through  coding  and  checkout 
Detail.  Total  effort  for  a  CPCI 


Table  1  [Cont)  Summary  of  Model  Characteristics 


2-6 


PRICE  S 


STRUCTURE 

Type.  Heuristic 

First  estimate.  Portion  of  development  cost  (design  cost) 
Multi -parameter 

Subsequent  estimates.  .  Functional  relationships 


Cost  is  related  to  predictor  variables  by  Tables  and  equations  that 
are  either  subjective  or  empirically  derived. 


Cost  and  effort  are  related  by  cost  per  unit  time  values  that  are 
constant  for  a  given  phase. 


OUTPUTS 


Cost.* 

Scope.  Detailed  Design  through  Installation 

Detail.  Three  phases,  Design  Implementation  Test  and 
Installation.  For  each  phase  by  activities 
system  analysis,  programming,  documentation, 
management,  quality  assurance.  Model  options 
include  independent  V&V,  system  integration. 

Time. 

Computer  units. 


*  Alternative  outputs  are  manhours  or  manmonths. 


Table  1  .Cont)  Summary  of  Model  Characteristics 
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TECOLOTE 


STRUCTURE 

Type.  Regression 

First  estimate.  Development  effort. 

Single  parameter 

Subsequent  estimates.  .  No  further  breakdown  of  effort. 

Development  effort  is  calculated  using  a  cost  estimating  equation  with 
number  of  instructions  as  the  independent  variable. 

OUTPUTS 

Effort. 

Scope.  Requirements  through  Operational  Demonstration 
Detail.  System  or  CPCI  level. 


Table  1  (Cont)  Summary  of  Model  Characteristics 
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SLIM 


STRUCTURE 


Type.  Phenomenological 

First  estimate.  Development  cost. 

Mul ti -parameter,  (linear  programming) 

Subsequent  estimates.  Allocations  using  fixed  ratios 


Effort  is  related  to  predictor  values  using  the  “software  equation. 
This  along  with  constraints  on  time,  effort  and  cost  define  a  range 
of  acceptable  solutions  (if  any). 


Cost  and  effort  are  related  by  a  constant  value  of  cost  per  unit. 


OUTPUTS 


Effort. 

Scope.  Detailed  design  through  installation  for  the 
primary  output.  Additional  outputs  include 
analysis  effort. 

Detail.  System  level 

Time. 

CPU  Time. 

Documentation. 


Table  1  (Cont)  Summary  of  Model  Characteristics 
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1 


WOLVERTON 


STRUCTURE 


Type. 

First  estimate. 
Subsequent  estimates. 


Heuristic 

Development  cost. 

Multi -parameter 

Allocation  using  fixed  ratios 


Cost  is  related  to  routine  size  and  category  by  a  constant  cost  per 
instruction  for  each  category  of  software. 


OUTPUTS 


Cost. 

Scope.  Analysis  through  Operational  Demonstration 

Detail.  Seven  phases,  each  with  up  to  25  activities  an 
eighth  phase.  Operations  and  Maintenance  has 
allocations  among  the  25  activities,  but  there 
is  no  guidance  for  allocating  the  eighth  phase 
from  the  total . 

Computer  cost. 


Taole  1  (Cent)  Summary  of  Model  Characteristics 
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3  EVALUATION  CRITERIA 

The  Air  Force  needs  reliable  procedures  for  estimating  software 
costs  to  support  its  activities  as  the  manager  of  weapon  system  development. 

It  is  necessary  when  examining  methods  for  making  cost  estimates  to  be 
mindful  of  the  Air  Force's  perspective  as  the  system  development  manager. 

The  Air  Force  does  not  develop  system  components  itself.  When  estimating 
the  cost  of  developing  and  operating  a  new  system,  it  must  at  first  consider 
industry-wide  capabilities  as  represented  by  experience  with  similar  weapon 
systems.  This  representation  of  development  performance  is  adequate  for  ' 
conceptual  studies,  but  it  is  not  valid  for  evaluating  proposals  for  specific 
subsystems  to  be  built  by  specific  organizations.  For  example,  a  single 
organization  may  obtain  good  results  using  a  given  method  of  cost  estimating; 
but  it  must  be  recognized  that  many  variables  such  as  experience,  support 
facilities,  and  management  techniques  are  relatively  fixed  in  that  organization. 
Their  influence  on  any  estimates  made  by  that  organization  are  minimal. 

However,  if  the  method  were  adopted  by  the  Air  Force  and  applied  to  many 
organizations  such  as  might  occur  in  a  major  weapon  system  development,  the 
results  may  not  be  satisfactory  at  all.  The  model  evaluation  was  designed 
to  look  at  software  cost  estimating  from  the  Air  Force's  point  of  view. 

The  choice  of  evaluation  criteria  was  affected  by  the  following 
considerations : 

•  A  number  of  different  software  cost  estimating  models  already  exist. 

•  Proponents  of  the  models  offer  testimonials  based  on  their 
particular  experience  and  estimating  needs. 

•  There  is  no  model  or  approach  that  is  not  without  both  supporters 
and  critics. 

•  Much  of  the  existing  literature  claims  there  is  no  reliable  method 
of  making  software  cost  estimates. 

Given  the  conflicting  evidence  it  seemed  reasonble  to  conduct  an 
evaluation  of  representative  cost  models  to  address  the  following: 
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t  The  needs  of  the  Air  Force  for  software  cost  estimates. 

•  The  extent  to  which  existing  software  cost  models  satisfy 
those  needs. 

•  The  characteristics  of  existing  model  structures  that  make 
them  good  or  bad  performers  for  Air  Force  purposes. 

•  Methods  for  improving  the  quality  of  future  Air  Force 
software  cost  estimates. 

The  evaluation  was  divided  into  two  parts: 

•  The  satisfaction  of  Air  Force  needs  for  software  cost  estimates 
in  terms  of  specific  items  of  information. 

•  The  realization  of  estimates  with  accuracy  acceptable  for 
making  decisions  concerning  selections  of  alternative  design 
concepts,  allocation  of  resources,  and  managing  the  software 
life  cycle. 

This  section  of  the  report  describes  how  criteria  were  defined  that 
establish  the  Air  Force  needs  for  cost  model  performance  in  terms  of  items 
of  information  and  accuracy. 

The  first  subsection  describes  the  Major  Weapons  System  Acquisition 
Process,  the  Software  Life  Cycle  and  the  Work  Breakdown  Structure.  It  then 
shows  how  these  lead  to  five  cost  estimating  situations  which  are  described 
in  terms  of  scope  of  the  life  cycle  addressed,  level  of  detail  in  the 
estimates  and  the  desired  estimating  accuracy. 

The  second  subsection  establishes  the  criteria  for  measuring 
estimating  accuracy;  and  the  final  subsection  discusses  some  evaluation 
criteria  that  were  considered  but  not  included. 

3.1  INFORMATION  NEEDS 

The  Air  Force  identifies  two  types  of  computer  system  development. 
One  is  the  creation  of  computer  systems  that  are  end  products.  That  is. 
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they  perform  a  separate  function.  These  are  for  the  most  part  management 
information  systems.  The  other  type  of  computer  system  is  an  integral 
part  of  a  larger  system.  It  is  characterized  by  stringent  and  complex 
interfaces  with  its  environment.  These  are  usually  called,  "embedded 
systems . " 

For  the  purpose  of  this  evaluation,  the  needs  for  software  cost 
information  will  be  established  by  the  process  governing  the  development 
of  embedded  software.  However,  this  should  not  limit  the  applicability 
of  the  results.  For  one  thing,  most  of  the  models  are  used  for  both 
types  of  development;  and,  for  another,  the  software  portion  of  the  develoo- 
ment  cycle  is  nearly  the  same  for  both  types  of  systems.  The  embedded 
system  development  must  be  governed  in  addition  to  its  own  requirements 
by  the  needs  of  the  weapon  system. 

The  representation  of  both  the  software  life  cycle  and  its  controlling 
environment,  the  weapon  system  life  cycle,  allows  us  to  specify  the  needs  for 
software  cost  estimates  considering  the  points  of  view  of  the  weapon  system 
and  the  software  components.  The  weapon  system  manager  must  know  how  the 
needs  of  the  software  components  will  affect  the  cost,  schedule  and  risk 
of  the  weapon  system.  He  must  also  know  how  the  performance  of  the  weapon 
system  in  terms  of  functions,  speed,  reliability,  etc.  are  affected  by  the 
software  system  cost,  schedule,  and  risk.  When  the  software  resources  have 
been  allocated,  the  software  subsystem  manager  must  assess  his  cost,  schedule, 
and  risk  in  terms  of  lower  level  design  choices.  He  as  well  as  the  weapon 
system  manager  must  make  preliminary  cost-performance  trade-offs,  prepare 
statements  of  work,  evaluate  proposals,  and  monitor  contracts. 

The  following  sections  describe  the  evolution  of  the  weapon  system 
definition  as  it  occurs  during  the  Major  Weapon  System  Acquisition  Process. 

The  aspects  of  the  weapon  system  that  establish  the  software  requirements 
are  highlighted.  The  software  life  cycle  is  presented  along  with  the 
definition  of  the  characteristics  that  contribute  to  the  estimation  of  its 
cost,  schedule,  and  risk. 
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The  Acquisition  Life  Cycle  for  Major  Defense  Systems  is  the  formal 
decision  process  regulating  the  acquisition  of  electronic  systems  that 
include  software.  Electronic  Systems  are  one  of  seven  types  of  system 
identified  in  MIL-STD-881A,  Work  Breakdown  Structures  for  Defense  Material 
Items  [34].  The  acquisition  of  computers  and  software  that  are  embedded  i 
a  weapon  or  command  and  control  system  are  normally  governed  by  the  Air 
Force  800  series  of  regulations. 

AFR  800-2  defines  the  Acquisition  Life  Cycle  for  Major  Defense 
Systems  as  normally  comprising  five  sequential  phases  (Figure  1): 

•  Conceptual 

•  Validation 

•  Full-Scale  Development 

•  Production 

•  Deployment 

Review  by  the  Defense  Systems  Acquisition  Review  Council  (DSARC) 
normally  follows  each  of  the  first  three  phases  and  Secretary  of  Defense 
approval  is  required  to  proceed  from  one  phase  to  another.  There  is  some 
flexibility  in  the  composition  of  the  phases.  In  general  the  process  is 
designed  to  insure: 

•  Continuing  operational  need 

•  Adequate  system  performance 

•  Acceptable  cost 

•  Favorable  cost  effectiveness  relative  to  other  alternatives 

A  Decision  Coordinating  Paper  (DCP)  is  prepared  to  support  each 
DSARC  review. 

The  procedure  used  as  the  basis  for  the  definition  of  need  is  taken 
from  [34]  which  is  based  mainly  on  interpretations  of: 
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AFSCP  800-3 
AFR  800-14,  Vol .  II 
AFR  800-2 
DoDI  5000.2 

Summary  of  the  Development  Phases. 

Conceptual  Phase. 

1.  Explore,  formulate,  and  evaluate  possible  system  requirements. 

2.  If  necessary,  devise  an  optimum,  affordable,  and  cost  effective 
preferred  approach  to  the  system's  development,  production, 

and  deployment. 

Considerable  preliminary  design  and  analysis  of  software  may  be  necessary 
to  support  these  objectives.  Demonstration,  prototype  and  simulation 
software  may  be  required.  Conceptual  Phase  design  and  analysis  should  be 
limited  to  whatever  is  necessary  to  establish  technical  feasibility  and 
credible  estimates  of  costs  and  development  times.  Design  and  analysis 
should  be  most  detailed  where  technical  risk  is  greatest. 

The  Conceptual  Phase  has  no  prescribed  time  limit.  Before  DSARC 
review  of  the  draft  DCP  begins,  the  program  can  be  terminated  with  the 
approval  of  the  highest  command  level  which  authorized  it.  Once  DSARC 
review  begins,  the  Conceptual  Phase  will  normally  end  with  the  Secretary 
of  Defense's  Program  Decision  to  proceed  into  the  Validation  Phase  (with 
or  without  specific  redirection),  or  to  end  the  program. 

Validation  Phase. 

1.  Assess  the  preferred  design  approach  selected  during  the 
Conceptual  Phase  by  comparing  it  with  the  Initial  System 
Specification, 

2.  Rectify  any  difficiencies  or  develop  a  new  approach  if  necessary. 

3.  If  and  when  a  sound  system  design  approach  is  achieved,  provide 
sound  technical,  contractual,  economic,  and  organizational  bases 
for  the  Full-Scale  Development. 
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Most  Validation  Phase  work  is  to  demonstrate  the  feasibility 
of  doubtful  components  and  subsystems  and  interface  definitions,  and  to 
improve  estimates  of  performance  cost  and  schedule.  All  can  be 
considered  risk-reduction  measures. 

The  Validation  Phase  may  also  include  contracted  design  competitions 

The  Validation  Phase  is  intended  to  reduce  risk  significantly  and 
to  allow  negotiation  of  clear  contracts  for  the  subsequent  acquisition 
phases.  The  development  of  unambiguous  specifications  and  testable  require 
ments  is  most  important. 

Full-Scale  Development. 

1.  A  working  prototype  of  the  system  {or  the  system  if  there  are 
no  repl i cas ) . 

2.  Test  results  proving  that  this  prototype  can  meet  its  functional 
and  performance  requirements. 

3.  A  Cadre  trained  in  the  system's  operation  and  maintenance. 

4.  The  documentation  needed  to  begin  the  system's  Production 
Phase  (if  any)  or  otherwise  needed  for  its  Deployment  Phase. 

For  the  system's  software  the  Full-Scale  Development  Phase  is 
intended  to  yield  the  initial  operational  versions  of  the  computer  programs 
not  prototypes. 

Tne  system's  operational  software  (i.e.  the  executives  and  applica¬ 
tions  programs  necessary  to  meet  the  system's  operational  requirements), 
ulus  the  support  software  necessary  to  build  and  maintain  the  operational 
software  and  to  support  the  Design,  Test,  and  Evaluation  and  Initial 
Operational  Test  and  Evaluation  functions  must  normally  be  completed 
during  Full-Scale  Development. 
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If  proprietary  software  is  to  be  incorporated  into  the  system,  the 
Government  must  decide  whether  the  price  represents  an  advantage 
over  contracted  development. 

Production  Phase. 

Activities  are  limited  to  maintenance  and  modification  of  existing 
software.  They  may  also  include  site-specific  testing  and  installation. 

Software  has  a  life  cycle  of  its  own  (Figure  2)  that  exists  in 
concert  with  the  weapon  system  life  cycle.  Software  requirements  for 
embedded  subsystems  are  established  primarily  by  the  needs  of  the  Weapon 
System. 


Table  2  [34]  describes  the  activities  and  products  comprising  the 
Software  Life  Cycle. 

The  functions  assigned  to  the  software  comprise,  along  with  the 
definition  of  the  computer  elements,  the  basis  for  estimating  the  time, 
effort,  and  other  resources  required  to  create  the  software  and  test  it. 

If  the  investment  needed  to  provide  the  prescribed  software  functions  are  not 
acceptable,  then  a  redefinition  of  the  allocation  of  functions  among 
hardware  and  software  may  be  necessary.  If  this  doesn't  resolve  the 
conflict,  it  may  be  necessary  to  revise  the  requirements. 

This  iteration  between  software  requirements  and  feasibility  is 
continuous  throughout  the  development  phases.  Problems  thought  solvable 
during  the  Concept  Phase  may  later  prove  not  to  be.  Sometimes  the  software 
definition  and  design  process  must  go  on  for  some  time  before  negative 
re'  are  obtained. 

development  of  systems  that  contain  software  is  an  iterative 
proci  the  steps  of  the  software  life  cycle  are  an  integral  part  of 

the  system  life  cycle.  Figure  3  [35]  describes  the  combined  system-software 
life  cycles. 
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TABLE  2  SOFTWARE  LIFE  CYCLE  ACTIVITIES  AND  PRODUCTS 


ANALYSIS  PHASE 


Activity 


Product! 


Devise  &  analyze  alternatives  A.l. 

for  the  system.  Segment  (if  2. 

any),  or  any  Software  Subsystem 
directly  containing  the  Computer 
Program. 

Allocate  requirements  to  B.l. 

the  Computer  Program:  i.e.. 

Functions.  2. 

Performance  (e.g.,  response 
times).  3, 

Interface  (with  others). 

Design  Constraints  (e.g., 
prescribed  algorithms,  core 
&  processing  time  budgets). 
Testing. 

Conduct  PDR(s)  for  the  C. 

Computer  Program's  CPCI(s). 


Tradeoff  study  reports 
Initial  or  Authenticated 
System  Specification  & 
Segment  Specification 
(if  any) . 

Authenticated  Development 
Specification  for  each  CPCI. 
Possible  higher-level  speci¬ 
fication,  and  ICD,  changes. 
Parts  of  draft  Product  Speci¬ 
fications  containing  design 
approaches  for  each  CPCI. 


PDR  minutes  and  action  item 
responses. 


DESIGN  PHASE 


Acti vitv 


Product! 


Define  a  1  gori thms  not  pre-  A.l. 

viously  prescribed.  2. 

Design  data  storage  structures.  3. 
Define  Computer  Program  logic.  4. 


Allocate  Computer  Program 
requirements  internally 
(e.g. ,  to  CPCs) 


Test  Planning. 


CDR(s)  for  the  Computer 
Program's  CPCI(s). 


Functional  flowcharts. 
Detailed  flowcharts. 

Data  Format  descriptions. 
Description  of  algorithms 
not  previously  prescribed. 

Preliminary  Product  Specifi¬ 
cations,  including  the  above. 


System,  Segment  (if  any) 
and  CPCI  Test  Plans. 

Preliminary  CPCI  Test  Procedures. 

CDR  minutes  &  action  item  responses. 
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TABLE  2  (Cont)  SOFTWARE  LIFE  CYCLE  ACTIVITIES  AND  PRODUCTS 


CODING  AND  CHECKOUT  PHASE 


Activity, 


Coding. 

Limited  checkout  of  compiler 
or  assembly  units. 

Corresponding  logic  &  data 
structure  revisions. 


TEST  AND  INTEGRATION  PHASE 


Product (s) 
A-8.  Code. 


Altered  Product  Specifications, 
including  compiler/assembly 
listings. 


Activity 


Test  Planning. 


Product(s) 

A.l.  Final  CPCI  Test  Procedures. 
2.  Segment  (if  any)  and  system- 
level  Test  Procedures, 


B.  ,  Module  tests.  B-O.l.  Test  Reports. 

2.  Computer  Program  coding 
changes. 

CPCI  tests  (PQT  &  FQT).  3.  Modified  Product 

Specifications. 

4.  Possible  high-level  specifi- 
Software  Subsystem  integration.  cation,  and  ICO,  changes. 


CPCI  tests  (PQT  &  FQT). 


INSTALLATIQN  PHASE 
Activity 

A.l.  DT&E  of  any  Segments, 
2.  System- level  DT&E. 


B.  Site  Adaptation  (if  any), 


Product (s) 

A. l.  Segment  (if  any)  Test  Reports. 

2.  System-level  DT&E  Test  Reports. 

3.  Computer  Program  coding 
changes . 

4.  Modified  Product  Specificaitons. 

5.  Possible  higher-level  specifi¬ 
cation,  and  ICD  changes. 

B. l.  Possible  site-specific  coding 

changes.  If  so: 

2.  Version  Description  Documents  & 

3,  Test  Reports. 


C.  lOT&E 


lOT&E  Test  Reports. 
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TABLE  2  (Cone)  SOFTWARE  LIFE  CYCLE  ACTIVITIES  AND  PRODUCTS 


OPERATION  AND  SUPPORT  PHASE 


Activity  Product (s) 


A. 

FOT&E 

A. 

Analogs  of  Test  and  Integration 
Phase  products. 

B. 

Construction,  instal lation,  & 
checkout  of  software  mainten¬ 
ance  &  training  facilities. 

B. 

Related  documentation. 

C. 

Software  maintenance  & 

C.l. 

New  software  Versions 

modi fi cation. 

2. 

Version  Description 

Documents. 

3. 

Possible  specificaiton 
changes. 

4. 

New  or  revised  Test  Plans 
and  Test  Procedures. 

5. 

Additional  tests. 

6. 

Additional  Test  Reports. 

Abbreviations 

CDR 

Critical  Design  Review 

CPC 

Computer  Program  Component 

CPCI  Computer  Program  Configuration  Stem 

DT&E  Development  Test  and  Evaluation 

FOT&E  Follow-On  Operational  Test  and  Evaluation 

FQT  Formal  Qualification  Test 

ICD  Interface  Control  Drawing 

lOT&E  Initial  Operational  Test  and  Evaluation 

PDR  Preliminary  Design  Review 

PQT  Preliminary  Qualification  Test 


Source; 


[34] 


Software  cost  models  must  contribute  to  a  rapid  determination  of 
economic  feasibility  of  the  software  conponents.  Ideally  a  model  will 
help  integrate  time,  effort  and  risk  in  order  to  establish  feasibility. 

It  will  do  this  using  information  describing  the  system  attributes. 

In  the  following  discussion,  the  weapon  system  acquisition  cycle 
is  used  to  define  the  cost  estimating  needs  and  the  software  life  cycle 
is  used  to  describe  the  elements  of  software  system  cost. 

Table  3  [34]  details  the  weapon  system  and  related  software  subsystem 
activities  comprising  rne  acquisition  life  cycle. 

Analysis  of  Table  3  'ndicates  a  continuous  transition  in  the 
needs  for  estimates  over  the  development  cycle.  During  the  early  phases 
the  need  is  for  high  level  or  aggregated  estimates  of  development  time 
and  cost  for  any  number  of  alternative  design  concepts.  As  the  system 
design  matures,  its  elements  become  defined  at  lower  levels  and  each  has 
a  greater  number  of  attributes.  The  individual  element  has  a  more  limited  ro'e 
in  the  system,  but  it  is  described  in  greater  detail  and  it  must  function  in 
concert  with  many  other  elements.  Initially,  we  might  speak  of  the 
navigation  subsystem  and  its  functions.  Later,  we  would  describe  the 
alignment  element  with  its  functions,  speed,  accuracy  and  interfaces  with 
the  accelerometers,  gyros,  etc.  Therefore,  inherent  with  the  process 
of  increased  system  definition  is  the  need  to  describe  levels  of  inter- 
gration  and  interface  in  addition  to  component  attributes. 

The  need  for  software  cost  and  resource  estimating  during  the 
development  life  cycle  proceeds  from  the  rapid  calculation  of  gross 
estimates  for  several  concepts  to  rather  detailed  estimates  devoted  to 
a  single  design.  Ideally  the  estim^iting  methods  needed  to  support  this 
process  would  be  functionally  oriented  in  the  early  phases. and  evolve  to 
variables  describing  design  characteristics  in  the  end  phases. 
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ESTIMATING  NEEDS  FOR  SOFfWARE  LIFE  CYCl.E  PHASES 


Work  Breakdown  Structure.  See  Appendix  B  for  definitions  of  levels. 


TABLE  3  (Con’t) 


The  need  for  precision  goes  through  a  similar  evolution.  During 
the  early  phases  it  is  only  necessary  to  determine'  if  a  concept  is  totally 
out  of  reach  in  terms  of  cost  or  development  time.  Subsequently  it  is 
necessary  to  weigh  the  cost  and  risk  of  one  design  concept  relative  to 
another.  The  final  estimates  involve  the  commitment  of  funds  and 
personnel  and  demand  the  greatest  possible  precision. 

Table  3  has  been  used  to  prepare  descriptions  of  five  cost 
estimating  situations  that  represent  the  different  kinds  of  estimates 
described  above.  These  descriptions  (Table  4)  which  include  the  scope 
and  detail  of  the  estimates  and  their  accuracy  are  the  basis  for  evaluating 
how  well  each  model  satisfies  the  needs  for  cost  information  during  the 
weapon  system  life  cycle. 

Having  described  the  general  need  for  cost  information  in  terms 
of  life  cycle  scope  and  detail,  it  is  possible  to  extend  the  criteria  in 
Table  4  to  include  the  Work  Breakdown  Structure  (Appendix  B).  This 
extension  provides  a  means  of  precisely  describing  all  the  software  estima¬ 
ting  needs.  Each  major  element  of  the  WBS  (Level  1)  is  divided  into 
appropriate  measures  of  the  software  product  (Level  2).  The  relationship 
between  the  two  levels  is  shown  in  Table  5. 

Appendix  B  describes  three  WBS  levels.  Table  3  indicates  that 
program  management  needs  extended  to  the  third  level.  However,  none  of 
the  models  evaluated  provide  cost  estimates  in  such  detail.  Carrying 
this  detail  in  the  evaluation  process  is  a  needless  complication  consider¬ 
ing  that  none  of  the  models  can  provide  the  information.  Therefore,  the 
third  level  of  the  WBS  is  considered  to  be  a  description  of  the  data  that 
should  be  included  in  the  higher,  level  estimates.  The  first  two  levels 
will  be  the  only  ones  considered  in  the  remainder  of  the  report. 

Figure  4  depicts  the  software  cost  elements  in  graphical  form.  The 
coTumns  represent  the  Software  Life  Cycle  phases  and  the  rows  represent 
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TABLE  4  FIVE  COST  ESTIMATING  SITUATIONS 


Conceptual  Phase,  Cost  Feasibility 
NEED 

In  support  of  the  analysis  of  perceived  deficiencies  in  existing 
systems,,  estimate  software  canponent  costs  and  development  times 
for  defined  alternatives. 

SCOPE 

Total  life  cycle  cost,  Conceptual  through  O&S. 

LEVEL  OF  DETAIL 

Weapon  System  -  Total  cost  of  all  software  components. 

INPUTS 

System  performance  and  functions. 

LEVEL  OF  PRECISION 
+  30% 

EXAMPLE 


The  software-related  costs  for  a  new  interceptor  aircraft. 


TABLE  4  ,Cont)  FIVE  COST  ESTIMATING  SITUATIONS 


2.  Conceptual  Phase,  Preliminary  System  Design  Studies 
NEED 

Support  the  evaluation  of  functional  allocations  for  system 
components  by  estimating  software  development  time  and  cost. 

SCOPE 

Cost  of  defining,  designing,  producing  and  owning  major 
software  components. 

LEVEL  OF  DETAIL 

System  functional  components. 

INPUTS 

System  segment  performance,  preliminary  performance  allocations 
preliminary  size  and  system  interface  descriptions. 

LEVEL  OF  PRECISION 
+  25% 

EXANPLE 

Compare  the  software  development  time  and  cost  for  a  four 
function  versus  a  five  function  navigation  system. 
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TABLE  4  (Cont)  FIVE  COST  ESTIMATING  SITUATIONS 


3.  Conceptual  Phase,  Preliminary  Contract  Cost  and  Schedule  Estimates 
NEED 

This  need  is  repeated  as  necessary  to  support  the 
evaluations  of  alternatives  leading  to  DCP  I.  The  level  of  detail 
remains  fairly  constant  although  some  analyses  may  require 
defining  critical  components  to  more  detail  than  the  others. 

The  only  thing  really  changing  is  the  confioence  in  the  results. 
System  components  are  defined  by  function  and  performance. 

SCOPE 

Validation  through  O&S. 

LEVEL  OF  DETAIL 

First  level  WBS  for  each  software  component 
INPUTS 

Software  functions,  performance,  interfaces,  inputs,  outputs. 

LEVEL  OF  PRECISION 
+  20% 

EXAMPLE 

Estimate  the  development  time  and  cost  for  a  real  time  display 
system  to  be  let  out  for  bids. 
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TABLE  4  (Cont)  FIVE  COST  ESTIMATING  SITUATIONS 


4.  Validation  Phase,  Support  of  Validation  Phase  Contracting 
NEED 

Allocate  funds,  support  RFP  preparations  and  assist  in  software 
related  proposal  evaluations  for  Validation  Phase  contracts. 

SCOPE 

Software  system  design  through  O&S. 

LEVEL  OF  DETAIL 

Software  WBS  level  2,  system  segment. 

INPUTS 

CPCI  characteristics  and  performance. 

LEVEL  OF  PRECISION 
+  15% 

EXAMPLE 

Estimated  cost  including  facilities,  training,  etc  for  the 
weapon  delivery  software  for  a  fighter-bomber. 
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TABLE  4  'Cone)  FIVE  COST  ESTIMATING  SITUATIONS 


5.  Full  Scale  Development,  Evaulate  Progress 
NEED 

Monitor  the  progress  of  software  system  components  during 
development. 

SCOPE 

CPCI  design  through  O&S 
LEVEL  OF  DETAIL 

Software  WBS  level  3,  CPCI  and  CPCR. 

INPUTS 

CPCI  and  program  functions  and  performance. 

LEVEL  OF  PRECISION 
+  10^ 

EXAMPLE 

Prepare  management  decision  boundaries  for  cost  and  schedul 
for  a  software  development  project  under  contract. 
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TABLE  5 

DECOMPOSITION  OF  SYSTEM  ELEMENTS  BY  MAJOR  WORK 
BREAKDOWN  STRUCTURE  DEFINITIONS 


LEVEL  2 


SYSTEM 

SEGMENT 

CPC  I 

CPRC 

LEVEL  I 

— 

, 

DEFINITION 

D 

bb 

! 

CODING 

:  X 

X  I 

DATA  CONVERSION 

1 

INFORMAL  TEST  Sc  INTEGRATION 

! 

X  i 

FOIJMAL  TEST  AND  INTEGRATION 

X 

X 

1 

1 

1 

INSTALLATION 

X 

DEVELOPMENT  FACILITIES 

X 

X 

TRAINING 

X 

X 

MANAGEMENT 

X 

X 

WBS  ELEMENT 


LIFE  CYCLE  PHASE 


ANALYSIS 


DESIGN 


CODING 

AND 

iCHECKOUT 


TEST  AND 
INTEGRATION 


O&S 


DEFINITION 


SYSTEM  * 

A 

SEGMENT* 

CPCfii 

COOING 


DATA  CONVERSION 

INFORMAL  TEST  AND  INTEGRATION 


CPCI 


CPRC 


EfUlODOO  DQDDQ 

BBBB 


CPCI 

CPRC 

FORMAL  TEST  AND  INTEGRATION 
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*  Definitions  of  these  terms  are  given  in  Section  3.1. 


Figure  4  The  Definition  of  the  System  Elements  and 
Their  Relationship  to  tne  Software  Life 
Cycle  and  WBS 
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the  elements  of  the  Work  Breakdown  Structure  (WBS).  The  WBS  elements  are 
identified  according  to  the  physical  decomposition  of  the  software  system. 

The  software  system  is  composed  of  the  following  elements; 

•  System  A  body  of  software  that  performs  an  identified  function 

in  the  weapon  system.  It  is  complete  and  distinguish¬ 
able  from  other  bodies  of  software. 

•  Segment  A  major  subsystem  or  component  of  a  system  usually 

identified  with  a  specific  function. 

•  Computer  Program  Contract  Item  (CPCI)  A  body  of  software 

identified  for  acquisition  by  separate  contract.  In 
large  systems  it  is  usually  part  of  a  Segment.  In 
smaller  systems  a  CPCI  may  be  equivalent  to  a  Segment 
or  even  a  System. 

•  Computer  Program  Reporting  Component  (CPRC)  In  large  systems  this 

represents  a  body  of  software  defined  for  purposes  of 
configuration  control  and  program  management. 

Figure  4  indicates  that  the  system  cost  elements  may  cross  life 
cycle  phase  boundaries.  It  is  important  to  depict  this  relationship  because 
many  cost  models  do  not  make  clear  distinctions  between  the  WBS  elements 
and  phase  costs . 

Figure  4  represents  a  detailed  template  for  depicting  the  estimating 
needs  represented  by  the  five  cost  estimating  situations  in  Table  4.  If 
the  cost  estimating  situation  calls  for  a  system  level  estimate  that  includes 
the  entire  life  cycle,  the  ideal  cost  model  would  be  shaded  at  the  system 
level  for  the  entire  row.  WBS  elements  stated  at  the  lower  levels  such  as 
Coding  and  Data  Conversion  would  be  cross-hatched  to  indicate  that  tne.v  are 
included  in  the  estimates  at  the  system  level  of  aggregation. 

In  Section  5,  Results,  Figure  4  is  used  to  describe  each  model', 
outputs.  In  that  section  a  summary  is  presented  that  indicates  how  well 
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the  model  satisfies  the  needs  associated  with  the  five  cost  estimating 
situations . 


3.2  ACCURACY 

Total  effort  was  selected  as  the  performance  measure  for  evaluating 
model  prediction  accuracy.  The  selection  was  made  because  of  it  is  relative¬ 
ly  easy  to  justify  and  interpret.  It  was  also  done  to  fix  attention  to  the 
single  model  output  that  everyone  should  agree  is  the  most  important  indi¬ 
cator  of  model  prediction  accuracy. 

It  is  possible  to  envision  several  alternatives  for  specifying  the 
accuracy  of  a  model  estimate.  For  example,  we  might  use  estimated  values 
of  the  costs  of  the  life  cycle  phases  to  construct  a  weigiTea  estimating  func¬ 
tion.  The  function  values  obtained  from  the  outputs  of  the  models  for  a 
given  project  would  be  compared  with^he  value  obtained  from  actual  measure¬ 
ments  to  produce  error  measures.  The  weights  in  such  an  approach  might 
also  be  obtained  from  the  test  data  sets.  Error  functions  could  also  be 
constructed  from  the  different  types  of  output  information  such  as  effort, 
computer  time  and  facilities. 

Total  effort  was  chosen  rather  than  cost  because  most  of  the 
models  being  evaluated  calculate  effort  and  because  the  available 
historical  data  are  in  terms  of  effort.  However,  the  use  of  effort  is 
desirable  also  because  it  avoids  the  need  for  adjusting  estimates  for 
variations  in  the  value  of  the  monetary  unit  and  the  problem  of  measuring 
overhead  and  indirect  costs.  These  items  vary  significantly  from  one 
organization  to  another. 

Unfortunately,  it  is  not  possible  to  specify  a  uniform  basis  for 
the  total  effort  measurement.  As  can  be  seen  in  the  Results  section,  the 
different  models  do  not  include  the  same  scope  of  life  cycle  activities 
in  their  estimates.  Therefore,  the  measurement  of  prediction  accuracy  had 
to  be  applied  to  the  primary  span  of  the  model  prediction.  In  most  cases 
this  means  the  prediction  that  is  constructed  using  the  primary  elements 
of  the  model  structure.  Some  models  use  these  primary  estimates  to 
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compute  other  phases  from  fixed  ratios.  We  believe  that  the  performance 
of  the  model  structure  is  better  represented  by  the  initial  estimate  in 
such  cases.  Having  selected  the  basis  for  measuring  model  estimating 
performance,  it  remains  to  define  the  way  to  use  the  measurement  to  obtain 
comparisons  among  the  models. 

Mean  Proportional  Error.  The  ratio  of  actual  to  estimated  project 
size  describes  the  error  as  it  relates  to  the  estimate.  It  is 
directly  transformable  into  the  percentage  error  of  the  estimate 
itself.  Being  a  proportion  it  allows  larger  errors  in  larger 
projects,  but  this  is  acceptable  because  we  tend  to  think  in  terms 
of  percent  error  rather  than  absolute  difference.  A  10  manmonth 
error  in  a  1000  manmonth  project  is  not  as  important  as  the  same 
error  in  a  6  manmonth  project.  The  disadvantage  of  the  MPE  as  it 
is  formulated  is  that  it  becomes  compressed  by  estimates  that 
are  large  compared  with  the  actual  value.  This  makes  the  standard 
deviation  small  when  taken  over  a  given  data  set.  To  reverse  the 
numerator  and  denominator  results  in  a  similar  weighting  when 
comparing  samples  containing  large  projects  with  samples  made  up 
of  small  ones. 

Average  Error.  The  average  difference  between  the  estimated  and 
actual  effort  taken  over  a  data  set  presents  a  measure  of  accuracy 
that  is  not  weighed  by  either  the  size  of  the  estimate  or  of  the 
actual  measure.  This  avoids  the  problem  associated  with  the  mean 
proportional  error,  and  dividing  the  average  error  by  the  mean 
project  size  in  the  test  sample  provides  scaling  for  the  measure. 

Root  Mean  Square  Error.  A  characteristic  of  some  software  cost 
models  is  their  tendency  to 'produce  estimates  that  are  very 
different  from  the  actual  experience  under  certain  conditions. 
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It  was  decided  to  select  an  error  measurement  scheme  that 
penalizes  such  extreme  behavior.  The  root  sum  square  error 
measure  provides  such  a  penalty  and  therefore  it  was  selected 
as  the  method  for  making  comparisons  among  the  models.  The  RSS 
error  is  divided  by  the  average  project  size  in  the  sample  set 
for  scaling.  The  measure  is  defined  to  be: 


RMSE  _ 

^  1=1 

2  1/2 

(ACT.  -  EST.)  ] 

ACT 

1  ^ 

Where: 

ACT.  = 

The  measured  size  of  the  ith 

project  in  the  sample  set. 

The  estimated  size  of  the  ith 

project. 

EST.  « 

N  = 

The  number  of  projects  in  the 

sample. 

3.3  OTHER  EVALUATION  CRITERIA 

In  addition  to  the  information  provided  and  the  prediction  accurac 
of  software  cost  estimating  models,  there  are  a  number  of  model  attribute 
that  would  influence  the  decision  to  select  one  model  structure  over  the 
others.  These  would  include: 

•  Data  needed  to  execute  the  model  . 

•  Effort  needed  to  execute  the  model . 

t  Time  required  to  obtain  estimates. 

•  Total  cost  of  estimates. 


Infor  lation  is  presented  that  would  allow  anyone  to  make  inferences 
regarding  a  model's  ranking  regarding  such  criteria.  However,  no  attempt 
was  made  to  compare  the  models  according  to  them  for  the  following  reasons; 

«  The  criteria  are  difficult  to  measure.  Any  weighting  of  the 
attributes  to  obtain  a  composite  score  would  be  arbitrary. 

•  Any  deficiencies  would  be  model  specific  and  the  evaluation 
is  concerned  with  the  performance  of  model  structures. 

^  A  model  that  provides  the  information  needed  and  does  it 
accurately  would  be  preferred  no  matter  how  badly  it  scored 
on  the  other  measures. 

The  findings  in  this  '^eport  should  indicate  how  well  one  model 
structure  performs  in  the  test  environments  compared  with  the  others 
evaluated.  If  two  models  satisfy  the  primary  criteria  equally  well,  it 
is  a  simple  matter  to  observe  the  other  attributes  that  may  be  important 
to  an  individual  organization  such  as  ease  of  execution,  data,  cost,  etc. 
These  considerations  would  have  different  importance  to  using  organizations. 
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4  EVALUATION  PROCEDURE 

The  following  steps  were  executed  in  performing  the  evaluation  of 
the  cost  estimating  models: 

1.  Select  models  for  inclusion  in  the  evaluation. 

Z.  Obtain  model  descriptive  materials. 

3.  Analyze  definitions  of  model  input  and  output  variables. 

4.  Prepare  model  descriptions. 

5.  Classify  models  by  type. 

6.  Prepare  list  of  input  and  output  variables. 

7.  Compare  model  outputs  with  established  evaluation  criteria 
for  needed  information. 

8.  Construct  test  data  sets. 

9.  Analyze  definitions  of  items  in  test  data  sets. 

10.  Establish  means  for  estimating  missing  input  data  items. 

11.  Prepare  input  data. 

12.  Execute  models. 

13.  Calculate  comparative  estimating  accuracy  according  to 
established  accuracy  evaluation  criteria. 

Several  steps  presented  problems  or  require  some  explanation: 

•  Definitions  of  model  and  data  set  variables. 

•  Model  types. 

•  Test  data  sets. 

•  Missing  input  data  items. 

4.^  DEFINITIONS  OF  MODEL  AND  DATA  SET  VARIABLES. 

A  problem  that  continues  to  limit  the  development  of  accurate 
software  cost  models  is  the  lack  of  good  quality  data  with  which  to  test 
theories  describing  the  relationships  between  cost  and  predictor  varia¬ 
bles.  An  important  aspect  of  data  quality  is  the  enunciation  ana  consis- 
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tent  application  of  definitions  of  the  variables  describing  the  software, 
its  development  environment,  and  its  performance.  A  substantial  amount 
of  effort  was  spent  during  the  evaluation  of  the  models  to  minimize  the 
adverse  effects  arising  from  discrepancies  in  data  definitions.  The 
following  paragraphs  will  examine  in  detail  two  important  model  variables: 
software  size  and  development  effort. 

Size  Definitions.  Eight  of  the  nine  models  included  in  this  evaluation 
use  size  as  an  input,  yet  problems  occur  when  trying  to  determine  precisely 
what  a  value  of  the  size  attribute  represents.  Often  researchers  either 
through  oversight,  lack  of  precise  data  or  ignorance  of  the  problem  fail 
to  specify  the  size  measure  completely,  ‘‘or  example,  number  of  source 
statements  and  number  of  object  instructions  are  two  of  the  terms  ihat 
are  frequently  presented  to  software  developers  in  questionnaires  without 
further  qualification.  As  a  result  it  is  possible  to  obtain  historical 
data  that  are  internally  inconsistent  by  100  percent  because  of  the  vague 
definitions.  Obviously  if  a  model  is  being  used  for  predictions  ana  the 
inputs  are  off  by  such  an  amount  the  estimates  will  be  similarly  affected. 
It  is  likely  that  in  many  cases  neither  the  person  supplying  the  historical 
data  nor  the  cost  analyst  realize  there  are  differences  in  interpretation 
of  the  data. 

Consider  the  deliberation  of  a  programmer  who  is  being  asked  the  size  ot 
one  of  his  programs  by  questionnaire.  The  question  is;  "Number  of 
Source  Statements."  If  the  programs  are  written  say  in  FORTRAN,  the 
compiler  will  normally  give  a  count  of  the  number  of  lines  in  the  program 
which  in  most  cases  will  be  equal  to  the  number  of  statements.  Most 
FORTRAN  compilers  limit  one  statement  per  line.  If  the  statement  is 
spread  over  several  lines  for  c’arity,  the  compiler  still  counts  ’t  as 
one  line.  For  the  most  part  the  FORTRAN  programmer  has  a  ready  source  fcr 
his  response  to  the  question. 


At  the  other  end  of  the  scale  is  the  programmer  writing  in  a 
freely  structured  language  such  as  COBOL.  His  response  is  considerably 
more  difficult.  Such  language  structures  use  punctuation  to  delimit 
statements  and  therefore  a  line  of  code  may  have  several  statements  or  a 
statement  may  stretch  over  several  lines.  Since  COBOL  compilers  do  not 
usually  indicate  the  number  of  statements  in  a  program,  but  only  the 
number  of  lines,  the  programmer  has  no  easy  source  for  the  requested 
information. 

Theconscientlous  programmer  may  make  this  problem  known  to  the 
cost  analyst  who  may  or  may  not  be  In  a  position  to  address  It.  More 
often  the  programmer  will  assume  the  question  calls  for  lines  of  code  or 
he  may  make  some  arbitrary  judgement  about  the  relationship  between 
statements  and  lines.  In  either  case  there  is  considerable  opportunity 
for  error  in  the  capture  of  the  most  commonly  used  predictor  of  program 
development  cost. 

There  are  other  problems  in  interpreting  the  term  “source  code 
state 'ser-ts"  as  the  descriptor  of  program  size. 

Most  higher  order  languages  permit  the  inclusion  of  comment 
statements  throughout  the  source  code.  These  statements  usually  describe 
what  the  program  is  doing  at  various  points.  Some  programmers  write 
many  comments.  Large  programs  exist  in  which  there  are  twice  as  many 
comment  lines  as  code  lines.  Other  programmers  do  not  write  any  comment 
lines.  Even  within  a  single  group  a  large  variation  exists  am,ong 
programmers.  A  programner  who  normally  comments  a  program  extensively 
may  do  so  very  sparingly  if  he  is  being  pressed  to  complete  the  program 
on  a  tight  schedule. 
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Some  cost  models  treat  coimients  the  same  as  lines  of  code; 
others  specifically  exclude  them.  If  asked  to  measure  his  program  without 
comments,  the  programmer  can  only  estimate.  If  he  has  the  time  and  inclin 
ation,  he  may  sample  parts  of  his  program  to  get  proportions  of  comment 
lines  to  code  lines;  or  he  may  guess.  Either  way,  additional  error  is 
introduced  into  the  measure. 

Data  specification  statements  are  eliminated  from  size  estimates 
for  some  models.  As  in  the  case  of  the  comment  lines,  the  proportion 
of  specification  statements  may  vary  substantially  from  program  to  pro¬ 
gram.  There  is  no  ready  way  to  count  the  number  of  specification  state¬ 
ments  in  FORTRAN.  By  comparison,  COBOL  groups  these  together  for  easy 
access. 


Compilers  usually  make  it  easy  to  share  source  code  among  programs 
Often  code  is  stored  in  libraries  and  automatically  called  in  by  the 
compiler  at  the  time  of  complilation.  Some  compilers  count  the  copied 
lines  in  addition  to  the  rest  of  the  program  lines;  other  compilers 
count  only  the  lines  calling  the  library  code;  and  still  other  compilers 
count  both  sets  thereby  producing  two  line  counts.  Unless  the  cost 
analyst  has  specified  how  copied  code  is  to  be  counted,  the  responses 
among  different  programmers  will  not  be  consistent. 

The  number  of  object  instructions  associated  with  a  program 
written  in  a  higher  order  language  has  at  least  as  many  possible  inter¬ 
pretations  as  the  number  of  source  statements. 

When  a  program  is  written  in  a  higher  order  language,  the  code 
produced  by  the  compiler  and  executive  support  programs  is  of  little 
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concern  to  the  programmer  unless  the  code  is  constrained  by  size  or 
speed  limitations.  But  even  if  the  programmer  is  faced  with  these 
problems,  normal  practice  is  to  work  with  the  higher  order  language  to 
satisfy  the  constraints.  The  tendency  to  work  with  the  higher  order 
language  and  the  decreasing  involvement  of  programmers  with  machine  level 
operations  mean  that  the  ‘information  describing  the  attributes  of  the 
program  in  its  executable  form  is  not  generated  during  compilation  and 
link  editing  or,  if  it  is,  it  is  ignored  or  even  not  understood. 

Given  the  inclination  toward  higher  level  language  use,  even 
if  the  definition  of  object  program  size  were  very  precise,  there  are 
many  programmers  who  would  not  know  how  to  respond  to  it  properly. 

Furthermore,  these  programmers  are  not  very  likely  to  be  frank  about 
ineir  ignorance. 

Probably  the  more  corimon  situation,  however  is  that  the  defini¬ 
tion  of  the  number  of  object  instructions  is  not  precise.  There  are 
several  ways  that  misinterpretations 'can  occur. 

Programs  written  in  a  higher  order  language  (or  in  assembly 
language  for  that  matter)  go  through  a  two-step  process  before  they  can 
be  executed  by  the  computer.  The  first  step  is  under  the  control  of  the 
higher  order  language  processor  and  it  produces  machine  level  code  that 
needs  other  code  before  it  can  be  executed.  This  code  is  sometimes  referred 
to  as  a  relocatable  module.  The  relocatable  module  is  processed  by  a  link 
editor  routine  that  produces  an  executable  program.  The  executable  program 
includes  all  the  routines  needed  in  addition  to  the  relocatable  module  and  it 
nas  an  integrated  addressing  scheme.  The  executable  module  is  the  program  that 
^.ds  needed  to  solve  a  given  problem.  But  it  can  easily  be  several  times 
'fic  size  of  the  sum  of  the  relocatable  modules  that  were  actually 
written.  Much  of  the  executable  code  is  taken  from  the  executive  library 
ind  performs  standard  mathematical  functions  or  drives  the  computer 
/stem  peripheral  equipment.  Furthermore,  some  computers  copy  the  basic 


4-5 


library  routines  many  times  and  insert  them  in  the  code;  others  share  the 
common  code  among  the  modules  without  duplicating  it.  Therefore,  the 
same  definition  of  program  size  can  elicit  very  different  responses  for 
a  program  written  to  the  same  specifications  because  of  the  way  the 
computer  executive  software  operates  or  the  way  the  programmer  directed 
the  link  editing  process. 

The  computer  memory  occupied  by  the  executable  program  contains 
not  only  the  instructions  but  also  areas  reserved  for  program  constants, 
data,  and  temporary  input-output  data  storage.  These  areas  in  total 
indicate  the  amount  of  memory  required  to  execute  the  programs.  The 
available  memory  may  be  a  constraint  or  otherwise  specified  by  the  user. 
Sometimes  the  size  of  the  executable  module  in  words  of  memory  is  usc-d 
synonymously  with  number  of  instructions.  These  two  measures  may  r? 
suDstantially  different. 

The  data  storage  area  size  may  be  very  small  comparec  w  th  the 
memory  area  containing  the  executable  .instructions  or  it  may  be  many 
times  as  large.  Therefore,  if  the  number  of  instructions  is  taken  to  be 
the  words  of  memory  occupied  by  the  data  and  instructions,  a  large  error 
may  resul t. 

Most  computers  do  not  incorporate  instructions  that  are  all 
same  size.  The  number  of  words  of  memory  required  to  contain  a  single 
instruction  may  range  from  one  to  several.  Some  estimate  of  the  average 
words  of  storage  per  instruction  must  be  made.  This  average  will  depend 
on  the  distribution  of  the  different  length  instructions  in  a  program 
and  would  vary  with  different  types  of  programs.  A  program  that  nas  a 
large  percentage  of  input-output  would  have  a  longer  average  instruction 
■than  one  in  which  logic  and  mathematical  computations  predominate. 

In  order  to  avoid  introducing  error  into  the  program  size 
definition  it  is  necessary  to  know  the  following  information; 
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«  Higher  Order  Language 

Whether  the  measure  is  statements  or  lines  of  code 
Whether  comments  are  to  be  included 
How  code  copied  from  another  source  is  to  be  counted 
How  data  specification  statements  are  to  be  treated 

•  Machine  Level  Language 

How  to  count  instructions 
Whether  data  areas  are  to  be  included 
Whether  relocatable  or  executable  code  measures 
are  wanted 

How  to  describe  the  library  or  copied  modules. 

Some  of  the  models  give  very  explicit  instructions  on  how  to 
describe  the  code  size  measure.  When  such  information  is  provided, 
we  can  describe  the  size  input  with  confidence.  Other  models  are  vague 
about  the  definitions  and  we  have  had  to  make  some  assumptions  about 
what  measure  is  appropriate. 

The  purpose  of  this  discussion  has  been  to  point  out  the  errors 
that  may  be  associated  with  the  use  of  the  size  measurement  in  software 
cost  estimating.  We  have  been  mindful  of  the  points  presented  above 
both  in  collecting  and  interpreting  the  data  used  to  evaluate  the  models 
for  this  study  and  in  preparing  inputs  for  the  different  models.  We 
have  read  model  descriptive  materials  and  when  possible  we  have  contacted 
the  authors  when  necessary  to  clarify  definitions.  When  it  was  necessary 
to  make  assumptions  about  either  the  data  or  the  inputs,  we  have  aes- 
cribed  them.  We  have  also  prepared  estimates  of  the  errors  asicclaieo 
with  possible  interpretations  of  size  measures.  Table  6  summarizes 
the  size  definitions  found  to  be  used  by  the  models  in  this  evaluation. 
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TABLE  6 


SIZE  DEFIiMITIONS 
USED  IN  THE  DIFFERENT  MODELS 


HIGHER  ORDER  LANGUAGE^^^ 

(0) 

WRITTEN 

EXECUTABLE 
DATA  DEFINITION 
COMMENTS 

ADAPTED 

EXECUTABLE 
DATA  DEFINITION 
COMMENTS 

COPIED  OR  TRANSFERRED  INTACT 
EXECUTABLE 
DATA  DEFINITION 
COMMENTS 

MACHINE  LEVEL  LANGUAGE^ 


WRITTEN 

EXECUTABLE 
DATA  DEFINITION 
COMMENTS 


ADAPTED 

EXECUTABLE 
DATA  DEFINITION 
COMMENTS 


COPIED  OR  TRANSFERRED  INTACT  U  I 

EXECUTABLE 
DATA  DEFINITION 
COMMENTS 


(1  Described  as  Lines  (L),  Statements  (S),  Unspecified  (U) 

(Z;  May  be  further  specified  as  Delivered  or  Not  Delivered 
(3)  Described  as  Number  of  Instructions  (I)  or  Number  of  Words  of  Storage  (W), 
Unspecified  (U) 

(4  Delivered  Code  Only 

(5;  Judgement  necessary  for  consistent  results 

(5'  Secondary  input  may  be  used  to  calculate  primary  input. 
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Development  Effort  Definitions.  Many  of  the  differences  between  the  stated 
needs  for  estimating  costs  and  the  outputs  of  models  are  semantic.  Models 
are  directly  or  indirectly  formed  from  experience  and  what  is  being  pre¬ 
dicted  depends  on  what  has  been  observed.  Past  data  is  seldom  the  product 
of  uniform  consistent  definitions  or  is  it  even  composed  of  the  same 
elements.  Furthermore,  there  is  no  common  set  of  definitions  that  are 
accepted  by  all  cost  analysts.  When  we  ask  different  cost  accountants 
what  is  included  in  a  data  set  under  program  design  cost,  we  will  get 
different  answers.  Even  if  the  answers  are  in  the  same  terms  there  is 
no  guarantee  they  are  consistently  interpreted  by  its  originators. 


Our  best  approach  is  to  consider  the  costs  on  a  relatively  high 
level  of  definition  (total  costs  as  opposed  to  subtotals).  Then  our  main 
concern  is  scope.  This  we  will  address  as  carefully  as  possible.  All 
this  is  said  not  to  be  negative,  but  to  present  an  accurate  picture  of 
the  conditions  under  which  all  cost  estimating  is  done.  We  as  an  industry 
have  established  a  certain  value  on' historical  cost  data  and  this  produces 
results  of  a  certain  quality.  As  long  as  the  results  obtained  with  this 
data  are  satisfactory,  there  is  no  reason  to  invest  resources  into  making 
it  better.  We  can  tell  if  the  presentation  of  the  data  is  consistent 
with  the  Air  Force  life  cycle  definitions  and  the  prescribed  Work  Break¬ 
down  Structure  (WBS),  but  when  totals  are  given  we  cannot  always  know  if 
the  elements  specified  are  actually  included  in  the  model  estimates. 

Different  elements  that  comprise  development  time: 

1.  Actual  at-the-desk  design,  coding  and  testing  hours. 

The  physical  direct  effort  required  to  produce  the  code. 

2.  The  time  charged  to  the  project  but  including  lost  time 
or  inefficiency  (breaks,  small  routine  administrative 
chores,  etc.) 

3.  The  time  not  usually  charged  to  the  project,  but  part  of 
every  job.  It  is  understood  that  in  order  to  realize  the 
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direct  effort  described  in  1.,  it  is  necessary  to  overstaff 
so  that  when  these  additional  hours  are  lost,  the  proper 
net  will  result.  This  last  category  includes;  sick  leave, 
vacation,  training,  other  scheduled  lost  time. 

Depending  on  the  method  used  by  an  organization  to  identify  how  personnel 
time  is  accounted  for,  different  portions  of  items  2  and  3  will  be 
associated  with  a  given  project. 

For  example,  consider  how  two  government  organizations,  one 
from  the  Air  Force,  the  other  from  the  Army  account  for  non-project  time. 

AIR  FORCE 

Hours  per  Month 


Mi  1  i  tary 

Civilian 

Holidays 

6.0 

6.0 

Leave 

6.9 

14.8 

Medical 

3.9 

6.8 

Education  and  Training 

3.8 

1.4 

Social  Actions, 

Organization  Duties,  etc. 

9.4 

1.0 

30.0 

30.0 

ARMY 

Non-Project  Training  -n 

Vacations 

Excused  Time 

Hoi idays 

Military  Leave  /  44  Hours  per  Month 

Sickness 

Non-Project  Meetings 
Special  Assignments 

, 

Assignments  to  Other  Projects 
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There  are  some  measurements  that  are  not  possible  to  interpret 
with  confidence  even  after  careful  analysis  of  the  sources  and  contacts 
with  the  model  developers.  Questions  would  have  to  be  asked  of  each 
respondent  or  originator  of  an  item  of  data.  We  can  only  list  the 
possible  errors  and  speculate  about  what  implications  the  probability 
of  their  existence  has  for  the  interpretation  of  the  results. 

4.2  MODEL  TYPES 

Our  objective  is  to  obtain  insights  into  the  relationships  between 
types  of  models  and  their  prediction  qualities.  Therefore,  it  is  desirable 
to  establish  some  method  of  characterizing  the  models  that  describes  their 
approach  to  making  estimates. 

We  have  selected  a  scheme  that  classifies  medel  structures  into 
three  types: 

•  Regression 

•  Heuristic 

•  Phenomenological 

The  following  discussion  explains  why  these  classifications  were 
selected  and  how  they  are  related  to  the  development  of  more  accurate 
model s . 

Classification  of  the  models  by  type  is  equivalent  to  forming  a 
hypothesis  about  the  structural  characteristics  that  affect  accuracy. 
Selecting  only  a  few  types  limits  the  consideration  to  the  major  struc¬ 
tural  attributes.  This  is  more  likely  to  provide  statistically  meaning¬ 
ful  results. 

At  the  detailed  level,  each  model  is  unique.  It  makes  its  est^m.ites 
using  different  parameters  and  procedures.  If  it  is  better  or  wor'^  ‘^lan 
another  model,  we  don’t  know  to  which  of  the  differences  between  tht  "cCflr, 
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to  attribute  the  performance.  On  the  other  hand,  if  we  hypothesize  that 
the  models  fall  into  certain  catagories  and  if  the  categorizations  are  re¬ 
lated  at  least  intuitively  to  accuracy,  then  the  measures  of  prediction 
performnace  can  be  associated  with  the  selected  attributes  and  model  develop¬ 
ment  can  be  directed  toward  structures  with  the  desirable  attributes.  The 
process  decreases  the  number  of  model  characteristics  that  are  considered 
to  contribute  to  estimating  performance.  If  the  results  indicate  no  rela¬ 
tionship  between  the  categories  and  prediction  accuracy,  then  the  hypotheses 
can  be  reseated  and  the  analysis  repeated. 

Top  down  and  bottom  up  are  not  sufficient  to  describe  models.  For 
example,  SLIM  estimates  the  development  cost  (which  is  not  the  same 
development  cost  estimated  by  the  Boeing,  Doty,  or  DoD  Micro  models)  for 
the  entire  system  then  extends  the  scope  of  the  estimate  to  include  the 
Requirements  and  Specifications  phases.  The  Design  and  Coding,  Integra¬ 
tion  and  Testing,  and  Installation  phases  are  fixed  portions  of  the 
Development  Phase.  Therefore  SLIM  at  least  in  one  of  its  operating  modes, 
is  a  top  level  estimate  constructed  from  individual  subsystem 
size  estimates.  That  initial  estimate,  however,  includes  only  a  part 
of  the  life  cycle  effort.  The  model  derives  higher  and  lower  level 
phase  elements  from  the  core  estimate. 

The  Dod  Micro  Procedure  describes  the  system  at  the  function  level 
and  assembles  the  weighted  components  into  a  net  development  effort  for 
the  entire  system.  The  net  development  is  extended  to  include  indirect 
effort  and  to  create  a  total  system  development  estimate.  This  is  then 
decomposed  into  phases  at  the  system  level. 

These  two  examples  illustrate  the  problem  of  trying  to  fit  models 
to  the  simple  descriptors  "top-down"  and  ”bottom-up."  It  is  necessary  to 
indicate  what  the  top  and  bottom  refer  to.  When  is  a  system  described  at 
the  Lop?  How  does  the  scope  of  the  life  cycle  included  in  the  estimate 
compare  with  the  level  of, detail  in  the  system  description? 
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We  have  tried  to  develop  a  method  of  describing  system  structures 
that  considers  these  questions.  By  following  a  line  of  reasoning  that 
utilizes  a  two-dimensional  description  of  software  life  cycle  effort  it 
is  possible  to  address  the  life  cycle  scope  and  system  level  of  detail 
independently.  This  is  an  important  distinction  that  has  not  been 
explicitly  addressed  by  most  model  developers  (Wolverton  and  PRICE  S  are 
two  exceptions).  Proceeding  in  this  manner  it  is  possible  not  only  to 
describe  models  more  accurately,  but  it  provides  an  insight  into  the  way 
the  different  models  proceed  from  inputs  to  outputs  that  is  the  basis  for 
categorizing  the  structures.  Hopefully  the  way  models  proceed:  the  method 
of  making  the  initial  estimate,  the  method  for  extending  or  detailing  the 
scope  and  the  way  of  developing  detail  can  be  associated  with  prediction 
performance. 

As  a  first  attempt,  the  categorizations  are  very  broad.  They 
concentrate  on  the  method  of  making  the  initial  estimate.  They  also 
indicate  the  general  method  of  making  t^he  subsequent  estimates.  If  we  look 
at  the  nine  models  included  in  this  study,  we  can  see  three  distinct  ways 
of  making  estimates.  These  are  described  in  the  following  paragraphs. 

The  Regression  Type  of  Model  Structure.  One  class  of  model  structures 
reflects  a  design  based  on  the  selection  of  the  life  cycle  element  of 
interest  (e.g.  life  cycle  effort,  development  effort,  or  coding  effort) 
and  a  hypothesized  relationship  between  the  element  and  a  number  of  selected 
inputs.  The  parameters  of  ti:e  hypothesized  relationship  are  obtained  by 
regression  and  the  model  becomes  a  single  cost  estimating  relationship  which 
is  treated  as  valid  for  whatever  population  is  believed  by  the  creator  and 
the  user  to  be  represented  by  the  data  used  to  calculate  the  parameters. 

The  data  may  be  stratified  in  some  way  thereby  producing  a  set  of  estima¬ 
ting  equations  with  each  member  of  the  set  applicable  for  the  estimating 
situation  described  by  the  stratification  parameter. 
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We  have  termed  models  structures  this  type  "Regression  Models" 
and  they  include  the  Aerospace,  Doty,  Farr  and  Zagorski,  and  Telecote 
models.  The  scopes  and  levels  of  these  models  differ,  but  they  share 
the  attribute  of  a  single  estimating  relationship  derived  by  linear 
or  log-linear  regression  using  various  inputs.  This  is  believed  to 
represent  a  recognizable  approach  to  model  construction  and  it  is  an 
objective  of  this  evalxiation  to  learn  if  the  approach  produces  more 
accurate  estimates  than  the  others. 

The  Heuristic  Structure.  Looking  again  at  the  mooels  under  investigation 
we  can  identify  another  approach  to  making  estimates. 

If  we  examine  the  Boeing  model,  for  example,  we  see  that  the 
system  is  divided  into  groups  of  code  that  are  characterized  by  type. 

Each  type  has  an  associated  productivity.  These  values  have  been  obtained 
from  historical  data  by  visual  curve  fitting,  regression  or  by  subjective 
assignment.  The  application  of  judgement  both  in  the  creation  of  tne 
procedure  and  in  the  establishment  of  parameters  is  typical  of  this  class 
of  model  structures.  A  system  level  estimate  is  calculated  by  sumning 
the  effort  for  the  different  code  groups.  The  system  level  estimate 
includes  the  entire  development  cycle.  This  is  divided  into  phases  using 
fixed  ratios.  Again,  the  ratios  may  be  obtained  by  objective  or  subjective 
means.  The  phases  are  adjusted  to  account  for  development  or  system- 
related  factors  that  are  believed  to  be  phase  dependent.  The  adjusted 
phase  values  are  added  together  to  produce  an  adjusted  development  effort. 

PRICE  S,  calculates  the  cost  of  the  Engineering  Design  life 
cycle  element  from  values  of  system  s'lZe,  resource,  ana  complexity. 
Adjustments  are  made  to  this  value  to  account  for  development  time, 
technology,  and  other  factors.  The  adjusted  value  is  used  to  calculate 
the  other  elements  of  the  system  development  cost  by  means  of  a 
cascading  technique.  The  final  time  distribution  of  system  cost  is 
subjected  to  further  adjustments  to  obtain  a  predetermined  time  phasing 
of  the  development  cost. 
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The  Boeing  and  PRICE  S  models  along  with  the  DoD  Micro  Procedure 
and  the  IJolverton  model  have  been  termed  "heuristic. "  The  dominant 
charactertst-ic,i)f  the  Heuristic  models  is  their  freedom  from  any  single 
mathematical  formulation.  .This  distinguishes  them  from  the  cost 


estimating  relationship  that  is  the  hallmark  of  the  Regression  type  of 


structure.  Heuristic  models  usually  combi ne  a  number  of  different  estima¬ 


ting  techniques.  The  calculation  of  the  estimate  usually  flows  through 


a  series  of  estimates  and  adjustments.  The  selection  of  the  individual 
steps,  the  cost  elements  treated  in  the  steps  and  the  method  of  determining 
the  adjustment  parameters  differ  significantly  among  the  models.  However, 
as  an  approach  to  making  software  cost  estimates  it  is  describable  and 
distinguishable  from  the  Regression  approach.  Measuring  the  differences 
in  estimating  accuracy  between  the  two  methods  would  give  considerable 
guidance  for  future  model  development. 


The  Heuristic  model  structure  combines  observation  and  interpretation 
with  supposition.  It  is  the  formal  representation  of  the  subjective  pro¬ 
cess  of  applying  experience.  Relationships  among  variables  are  stated 
without  justification  (e.g.  cost  per  pound  decreases  with  increasing  size, 
development  effort  is  related  to  number  of  file  formats,  the  number  of 
instructions  per  month  depends  on  the  type  of  code,  etc.).  Then  subjective, 
semi-empirical,  or  empirical  adjustments  are  made  to  the  base  estimate.  In 
some  of  the  models  included  in  this  evaluation  this  process  is  extremely 
compl icated. 

The  advantage  of  the  Heuristic  structure  is  that  it  does  not  have 
to  wait  for  the  establishment  of  formal  relationships  descrio’-ig  how  the 
cost-driving  variables  are  related.  The  process  of  model  development 
proceeds  intuitively.  As  situations  are  encountered  where  the  model  fails 
to  perform  acceptably,  an  adjustment  or  addition  is  made  and  the  process 
continues.  Over  a  period  of  time  a  given  model  can  become  very  effe^'.  .e 
in  a  stable  predicting  environment.  It  becomes  the  repository  for  tr. 
:ollected  experience  and  insights  of  the  model  designers  and  users. 
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The  Phenomenological  Model.  One  model  in  the  evaluation  group,  SLIM 
is  unique  in  that  it  incorporates  a  concept  that  is  explainable  in  terms 
of  a  basic  phenomenon  that  is  not  limited  to  the  mechanics  of  software 
development.  The  relationship  can  be  derived  in  terms  of  the  rate  of 
solving  problems. 

SLIM  uses  the  Ravleigh-Norden  function  [36]  [37]  [38]  to  describe 
Lne  time  distribution  of  effort  during  the  software  life  cycle.  It  has 
been  shown  [36]  that  this  function  represents  the  time  distribution 
of  effort  required  to  solve  a  given  number  of  problems  under  the  assumption 
of  a  constant  learning  rate.  The  ability  to  describe  observed  processes  in 
terms  of  elementary  phenomena  is  characteristic  of  the  more  mature  sciences. 
It  allows  complex  relationships  to  be  explained  by  interactions  among 
elementary  functions.  These  functions  are  verifiable  by  con¬ 
trolled  experiments. 

Although  it  may  be  argued  that  SLIM  incorporates  too  many  empirical 
adjustments  to  be  a  purely  phenomenological  model,  it  nevertheless  is  the 
only  model  to  use  an  observed  basic  relationship  to  make  estimates. 

The  difference  between  a  Phenomenological  model  and  a  model  based 
on  one  or  more  hypothesized  relationships  which  could  be  used  in  either  a 
Regression  of  Heuristic  structure  depends  on  the  source  of  the  hypotheses. 

If  the  hypotheses  are  motivated  by  tendencies  observed  among  the  variables 
describing  software  and  its  development  resources,  and  if  the  hypotheses 
are  describable  only  within  the  context  in  which  they  are  used,  then  the 
resulting  model  structure  is  not  phenomenological.  The  phenomenological 
model  must  incorporate  ideas  or  processes  that  can  be  observed  and 
measured  independently  of  the  software  development  process.  The  justifica¬ 
tion  can  be  derived  after  it  was  observed  in  a  software  context  and  a  model 
may  oecome  phenomenological  after  it  was  previously  classified  otherwise 
if  the  qualifying  condition  is  satisfied. 
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Given  the  present  situation,  where  no  generally  accepted  statement 
exists  of  the  elements  of  the  software  development  process,  the  true 
phenomenological  model  is  mostly  an  ideal.  But  the  search  for  basic 
understanding  and  description  must  be  made  if  we  are  to  obtain  real 
improvements  in  prediction  quality.  After  a  while  a  heuristic  model  may 
collapse  of  its  own  weight  as  it  tries  to  adapt  to  each  new  experience. 

The  phenomenological  model  ultimately  promises  the  simplicity  of  represen¬ 
tation  that  characterizes  scientific  laws. 

The  development  of  phenomenological  models  requires  the  explanation 
of  basic  processes.  Some  of  these  processes  are  being  investigated. 

Halstead  [39],  for  example,  has  explored  the  relationships  between  algorithms 
and  the  effort  needed  to  code  them.  Other  researchers  have  identified 
phenomena  related  to  the  development  of  systems  such  as  the  law  of 
increasing  entropy  [40],  life  cycle  phase  interrelationships  [41],  ripple 
effect  [42],  and  others.  These  elements  will  undoubtedly  contribute  to  the 
establishment  of  new  models  based  on  elementary  phenomena. 

Describing  the  Estimating  Process.  The  definition  of  the  model  type 
classifications  is  intended  to  establish  general  approaches  to  estimating 
that  are  associated  with  greater  or  less  prediction  accuracy.  Within  each 
general  type  of  structure  it  is  necessary  to  describe  more  detailed  aspects 
of  the  estimating  process. 

The  estimating  procedure  is  described  according  to  the  cost  element 
that  is  estimated  first  and  the  method  used  to  make  it  and  then  the  method 
used  to  obtain  subsequent  estimates. 

The  cost  element  used  for  the  initial  estimate  has  an  important 
bearing  on  the  level  of  aggregation  that  is  associated  with  the  greatest 
accuracy.  There  may  be  certain  combinations  of  the  level  of  the  first 
estimate  and  different  methods  for  making  it  that  have  implications  for 
accuracy. 
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Having  identified  the  best  way  to  obtain  the  initial  estimate,  it 
should  be  useful  to  investigate  how  subsequent  estimates  are  either 
expanded  in  scope  to  obtain  a  synthesis  of  the  entire  life  cycle  or 
decomposed  to  allocate  portions  of  it. 

The  classifications  used  for  describing  the  estimating  procedure 
are  as  follows: 

Level  of  Initial  Estimate 

System  Total  Development 

System  Analysis 

System  Design 

System  Coding 

System  Test 

CPCI  .Total  Development 

CPCl  Analysis 

CPCI  Design 

CPCI  Coding 

CPCI  Test 

Method  of  Making  Initial  Estimate 

Single  Parameter 
Multi  Parameter 

Method  of  Making  Subsequent  Estimates 

Cost  Estimating  Relationship 
Ratios 

No  Further  Decomposition 
4.3  TEST  DATA  SETS 

The  characteristics  of  each  test  data  set  are  d»-  .  .be  i  Appendix 
D,  Data  Preparation.  This  section  describes  the  environments  from  whi^n 
the  data  were  obtained  and  tells  how  they  were  obtained. 

The  data  used  to  evaluate  model  estimating  performance  was 
compiled  from  three  sources: 
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•  U.S.  Air  Force  Data  Systems  Design  Center  (USAF/DSDC) 

•  Goddard  Spaceflight  Canter,  Software  Engineering 
Laboratory  (GSFC/SEL) 

•  The  system  development  center  of  a  large  corporation 

The  data  sets  from  the  first  and  third  sources  were  obtained 
by  GRC  from  the  developing  organizations  using  questionnaires  and 
other  devices  that  will  be  cescribed  later.  The  second  data  set  was 
given  to  us  by  Prof.  V.  R  Basil i  of  the  Computer  Science  Department, 
University  of  Maryland.  The  University  operates  the  Software  Engineer¬ 
ing  Laboratory  under  a  grant  from  the  National  Aeronautics  and  Space 
Administration  (NASA). 

Air  Force  Data  Systems  Design  Center.  The  DSDC  develops  large,  standard 
data  systems  for  Air  Force  use  world-wide.  These  are  data  management 
systems  such  as  payroll,  logistics,  and  personnel  applications.  The 
programs  are  written  in  COBOL  and  are  developed  and  maintained  at  a 
single  site.  Under  two  separate  Air  Force  contracts,  (Electronic  Systems 
Division,  1975  [43]  and  Sacremento  Air  Logistics  Canter,  1978  [44])  GRC 
collected  data  describing  data  system  development  hours,  system 
characteristics  and  personnel  data.  From  these  data  17  projects  were 
selected  for  use  in  this  evaluation. 

Data  describing  the  hours  charged  by  individuals  to  each 
project  were  obtained  from  the  PARMIS  history  files.  PARMIS  was  a 
project  status  reporting  system  used  at  the  Design  Center  during  the 
years  1971  through  1978.  Hours  were  reported  by  project  staff  members 
on  a  weekly  basis.  The  hours  were  identified  according  to  project  and 
activity. 

GRC  wrote  a  computer  program  that  tabulates  the  hours  for  each 
system  according  to  a  standard  set  of  life  cycle  phases  and  activities. 
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A  questionnaire  was  given  to  project  leaders  who  provided 
information  describing  the  systems  and  the  experience  of  the  personnel. 

The  characteristics  of  the  programs  were  obtained  from  two 
different  sources.  The  command-level  systems,  which  are  implemented  on 
Honeywell  H-6000  computers,  were  processed  by  a  program  called  the 
Program  Profile  System  (PPS),  which  was  developed  at  the  Design  Center. 
The  PPS  analyzes  the  source  code  and  tabulates  the  number  of  lines, 
statements,  record  descriptions,  etc.  The  base-level  systems,  which 
are  implemented  on  Burroughs  B-3500  equipment  were  described  in  the 
materials  prepared  by  the  Air  Force  as  part  of  the  request  for  proposals 
to  replace  the  base-level  systems. 

Using  these  sources  and  with  considerable  help  from  the 
personnel  at  the  Design  Center,  a  rather  complete  and  reliable  set  of 
data  have  been  compiled  describing  the  systems. 

Goddard  Spaceflight  Center,  Software  Engineering  Laboratory.  The  SEL  is 
responsible  for  maintaining  a  high  quality  data  base  describing  the 
software  development  experience  at  the  Goddard  Spaceflight  Center.  A 
full-time  staff  collects  and  analyzes  data  describing  the  system 
attributes,  development  methodologies,  and  resource  expenditures.  The 
software  operates  on  large  ground-based  computers  in  support  of 
satellite  operations-  The  primary  language  is  FORTRAN  and  much  of  the 
code  operates  under  a  time  constraint. 

The  data  used  for  the  evaluation  of  estimating  accuracy  represent 
seven  systems.  Two  of  the  systems  are  partitioned  into  ten  subsystems. 

Commercial  Data.  GRC  has  an  arrangement  with  a  large  corporation  where¬ 
by  the  two  companies  exchange  Information  describing  software  develop- 
(nent  experience.  The  data  used  in  this  study  was  provided  by  the 
company's  central  system  development  facility.  The  applications  are 
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data  management  systems  written  in  COBOL  and  two  data  base  management 
languages.  The  facility  utilizes  modern  programming  practices  and 
provides  the  programmers  with  online  prograinning  and  debugging  capability. 

Data  describing  eleven  systems  were  tabulated  from  company 
records  onto  a  modified  version  of  the  questionnaire  used  to  capture 
the  Air  Force  Data  System  Design  Center  data. 

4.4  MISSING  DATA 

The  models  selected  for  evaluation  represent  a  range  of  different 
estimating  methods.  They  were  constructed  in  many  environments  and  as  a 
result  they  include  as  a  group  many  different  input  and  output  variables. 

It  was  demonstrated  in  the  last  section  how  two  of  the  most  common  model 
variables,  size  and  effort,  can  be  represented  by  over  a  dozen  different 
definitions.  Add  to  these  differences,  the  number  of  types  of  variables 
that  may  be  used  in  model  construction  and  it  is  easy  to  see  how  the 
requirements  for  test  data  variables  becomes  very  large.  The  nine  models 
in  the  test  group  require  more  than  thirty  variables  to  define  their  inputs 
and  outputs,  and  that  does  not  include  the  minor  variations  that  exist 
among  the  variables. 

It  was  not  possible  to  obtain  test  data  that  includes  all  the 
model  variables  and  their  variations.  The  compatibility  between  data 
availability  and  that  needed  to  execute  and  compare  model  outputs  is 
represented  in  Figure  5. 

The  Roman  numerals  identify  the  inputs  and  outputs  contained  in 
a  given  data  set.  A  data  set  may  contain  some  of  the  inputs  for  each  of 
the  models  (Greek  letters)  and  some  of  the  outputs,  but  all  the  models 
could  not  be  executed  and  compared  using  the  same  data. 

Our  approach  was  to  execute  each  model  according  to  the  outputs 
it  provides  and  the  inputs  needed  to  make  them.  If  a  model  is  designed 
to  predict  design,  code,  and  test  effort,  then  the  test  data  were  adjusted 
to  describe  these  portions  of  the  life  cycle. 
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Missing  inputs  were  obtained  using  estimating  relationships  con¬ 
structed  from  the  other  inputs.  For  example,  if  a  model  called  for  a 
number  of  object  instructions  and  a  given  data  set  '.ontained  only  lines 
of  source  code,  a  relationship  was  developed  from  other  available  data 
that  could  be  used  to  predict  number  of  object  instructions  from  lines 
of  source  code. 

Whenever  it  was  necessary  to  estimate  missing  input  data,  both 
the  expected  value  and  its  variance  were  estimated.  The  variance  is  used 
to  indicate  the  effect  of  the  uncertainty  in  the  estimate  in  the  evalua¬ 
tion  of  prediction  accuracy. 

The  input  data  for  each  of  the  models  are  presented  in  Appendix  d, 
Data  Preparation.  The  effect  of  missing  data  on  the  evaluation  are 
discussed  in  Section  6.0,  Analysis  of  Results. 


5  RESULTS 

The  cost  model  evaluation  criteria  (Section  3)  were  designed 
to  consider  two  aspects  of  the  state  of  the  art  in  software  cost  model 
construction: 

•  The  satisfaction  of  US  Air  Force  needs  for  software 
cost  information  (Section  3.1);  and 

•  The  satisfaction  of  US  Air  Force  needs  for  cost  estimating 
accuracy  (Section  3.2). 

The  objective  of  the  evaluation  of  the  first  aspect  is  to  measure 
how  well  existing  models  satisfy  Air  Force  information  needs  with  regard 
to  software  cost  items  and  their  levels  of  detail. 

The  evaluation  of  the  second  aspect,  accuracy,  relates  estimating 
accuracy  to  model  structures.  The  objective  is  to  identify  ways  of  making 
estimates  that  can  be  demonstrated  to  produce  greater  accuracy.  The 
objective  does  not  include  a  ranking  of  the  existing  cost  models  or  any 
general  statements  of  their  individual  estimating  accuracies.  The  results 
will  be  used  to  design  a  research  program  for  improving  the  accuracy  of 
software  cost  models. 

A  comparison  of  the  outputs  of  the  models  under  investigation  with 
the  Air  Force  estimating  needs  indicates  the  following: 

•  The  supporting  materials  for  most  of  the  models  do  not  clearly 
state  the  elements  included  in  their  estimates  and  are  not 
precise  about  their  definitions. 

•  The  existing  models  are  better  able  to  satisfy  information 
needs  early  in  the  acquisition  life  cycle. 

•  None  of  the  models  included  in  this  study  fully  satisfy  the 
Air  Force  need  for  information  either  with  regard  to  scope  or 
detai 1 , 
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t  The  models  tend  to  be  phase  oriented  and  do  not  properly  describe 
activities  that  cross  phase  boundaries.  This  precludes  obtaining 
data  compatible  with  both  management  planning  (phase  related)  and 
product  cost  (WBS). 

•  Although  most  of  the  models  use  the  summation  of  program  or 
module  sizes  to  make  their  cost  estimates,  only  one  model  studied 
provides  for  keeping  track  of  the  cost  on  a  component  basis  and 
accounts  for  the  cost  of  system  integration.  None  of  the  models 
provide  for  all  four  levels  of  system  definition  called  for  in 
the  Work  Breakdown  Structure  (Ref.  Appendix  B). 

Based  on  the  relative  root  mean  square  error  measure  of  performance; 

•  Recalibration*  is  the  primary  factor  contributing  to  !  r. 
differences  in  estimating  performance  among  the  models  tested. 

•  The  contribution  of  model  structure*  to  estimating  accuracy  is 
not  significant  when  the  models  have  been  calibrated  to  the 
development  environment*. 

e  The  development  environment  significantly  affects  the  relative 
performances  of  the  models  tested. 

•  The  effect  of  development  environment  on  estimating  performance 
precludes  the  possibility  of  obtaining  generally  applicable 
measures  of  model  performance  without  applying  additional  controls. 

•  Models  that  do  not  use  size  as  an  input  may  perform  as  well  as 
those  that  do. 

•  The  average  RMS  Error  for  all  tested  models  is  unacceptably 
large  for  Air  Force  estimating  purposes. 

•  The  use  of  models  that  are  not  calibrated  to  a  given  development 
environment  can  lead  to  very  large  estimating  errors. 

•  The  best  performance  obtained  by  any  group  of  the  models  tested 
is  not  adequate  for  Air  Force  needs. 

The  detailed  presentation  of  the  results  of  the  evaluation  of  the 
software  models  is  in  two  parts.  The  first  part  compares  the  outputs  or 

*  Definitions  of  these  terms  are  given  in  Section  1.5. 
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estimates  produced  by  the  models  with  the  needs  associated  with  the  major 
weapon  system  development  process.  The  second  part  shows  the  prediction 
accuracy  of  the  different  models. 


5.1  COMPLIANCE  WITH  AIR  FORCE  COST  INFORMATION  NEEDS 

The  description  of  each  of  the  models  was  studied  to  learn  exactly 
what  the  outputs  represent.  The  Air  Force  estimating  needs  (Figure  4) 
require  information  that  includes  certain  cost  components  (administrative 
expenses,  non-delivered  software  expenses,  overhead,  holidays,  etc.) 
presented  at  specific  levels  of  the  Work  Breakdown  Structure  (Appendix  B. 
Table  5) . 

The  following  paragraphs  describe  the  outputs  of  the  models  in 
terms  of  the  needed  information.  In  many  cases  it  was  not  possible  to 
determine  whether  the  information  being  sought  is  included  in  the  model 
estvnates.  Model  descriptions  are  often  vague  about  the  details  of  the 
outputs.  Sometimes  it  was  necessary  to  acquire  a  detailed  knowledge  of 
the  original  data  used  to  constri/ct  the  model  in  order  to  identify  the 
elements.  It  is  likely  that  precise  answers  to  some  of  the  questions  of 
definition  are  unknown  to  anyone  save  the  individuals  who  originally 
recorded  the  data.  In  many  cases,  data  were  not  recorded  consistently. 

Our  attempts  to  obtain  precise  definitions  of  model  estimates  and 
to  identify  the  included  cost  elements  indicates  that: 

•  The  supporting  materials  for  most  of  the  models  do  not  clearly 
state  the  elements  included  in  their  estimates  and  are  not 
precise  about  their  definitions. 

Our  approach  was  to  read  the  model  descriptions,  check  the 
assertions  with  the  original’  data  source  when  possible,  ask  questions 
of  the  model  creators  when  they  were  available  and  finally  to  draw 
our  own  experience.  Hopefully,  the  results  are  valid  descriptions  o- 
the  model  outputs . 
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Notice  that  the  figures  (Figures  6  through  14)  differentiate  between 
output  items  that  are  explicitiy  presented  (shaded  segments)  and  those 
that  are  included  as  parts  of  other  values  (cross-hatched  segments).  The 
numbers  in  parentheses  indicate  how  some  of  the  items  are  grouped  for 
presentation  by  the  model.  A  model  that  completely  satisfied  all  Air  Force 
cost  information  needs  would  have  all  the  segments  shaded. 

.Merospace  Corporation  ^  Figure  6).  The  presentation  is  typical  of  the 
regression  models.  System  level  estimates  of  cost  or  effort  are  obtained 
oy  fitting  linear  or  log-hnear  functions  to  historical  data.  The  elements 
believed  to  be  represented  in  the  total  are  indicated  by  the  cross-hatching. 
However,  since  the  historical  data  often  comes  from  several  sources,  the 
components  are  often  not  the  same.  Data  quality  is  usually  a  problem  so 
little  confidence  can  be  given  *0  any  description  of  the  elements  included. 

The  figures  show  only  those  data  items  explicitly  mentioned  in  the 
model  descriptions.  Therefore,  items  such  as  data  conversion  and  installa¬ 
tion  may  actually  be  included  in  the  Aerospace  Model  estimates  depending 
on  the  data  collection  and  tabulation  practices  governing  the  historical 
data  from  which  the  model  is  derived.  The  model  does  not  provide  any 
breakdown  of  the  system  level  estimate. 

•3oeing  Computer  Services  (Figure  7).  This  model  divides  the  system  level 
estimate  into  life  cycle  phases.  The  process  is  similar 
to  that  used  in  a  formal  manner  by  PRICE  S  and  informally  or  subjectively 
by  the  Wolverton  model.  In  all  three  models  the  effort  distributed 
among  the  phases  is  not  in  fixed  proportions.  In  the  case  of  the  Boeing 
model  adjustment  factors  reflecting  environmental  characteristics  are  phase- 
dependent.  Therefore,  the  recombination  of  the  phases  makes  a  total  that 
is  different  from  the  original  life  cycle  estimate.  PRICE  S  allows 
weighting  factors  to  be  applied  to  the  distribution  among  phases  and 
WoU'erton  allows  the  estimates  to  be  changed  to  accomodate  staffing  and 
schedule  constraints.  All  the  other  models  that  decompose  the  total  effort 
info  phases  do  so  with  fixed  ratios. 
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Notice  that  while  the  Boeing  model  includes  the  coding  effort,  only 
the  portion  that  occurs  during  the  Coding  and  Checkout  Phase  would  be 
specifically  identified.  Any  coding  done  during  the  Test  and  Integration 
Phase  would  be  represented  as  Test  and  Integration  effort.  A  similar 
misrepresentation  occurs  for  design  that  is  done  after  the  end  of  the  Design 
Phase.  This  problem  exists  in  all  the  models.  The  proper  identification 
of  the  elements  in  the  work  breakdown  structure  according  to  their  proper 
occurrence  in  the  life  cycle  will  not  be  possible  until  considerably  more  effc'^ 
is  applied  to  the  proper  identification  and  tabulation  of  data  describing 
the  software  development  process. 

DoD  ilicro  Estimating  Procedure  (Figure  8).  A  system  level  estimate  is 
obtained  by  a  weighted  count  of  program  functions.  The  estimate  is 
allocated  to  the  life  cycle  phases  using  a  fixed  distribution.  Training, 
management  and  other  indirect  activities  are  included  as  a  multiplier  of 
the  direct  effort. 

Ooty  Associates;  and  Farr  and  Zagorski  (Figures  9  and  10).  These  two 
models  were  derived  from  the  same  data  definitions.  The  data  used  in  the 
Farr  and  Zagorski  model  was  a  subset  of  that  used  in  the  Doty  model.  The 
estimates  are  made  at  the  program  level.  The  included  activities  begin 
with  the  detailed  program  designed  and  extend  to  release  of  the  programs  to 
integration  and  system  testing.  The  Doty  model  includes  an  estimate  or 
development  time. 

PRICE  S  (Figure  11).  By  properly  choosing  weighting  factors  for  the  three 
life  cycle  phases  and  five  activities  presented  by  this  model,  it  is 
possible  to  obtain  many  elements  of  the  desired  cost  information.  PRICE  o 
Is  unique  among  the  models  evaluated  in  that  it  allows  subsystem  level  defi¬ 
nitions  to  be  explicitly  presented  for  all  of  its  life  cycle  and  actis  r./ 
elements.  Adjustments  to  account  for  the  additional  effort  needed  to 
integrate  the  subsystem  into  the  system  may  be  specified  by  the  user. 
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Figure  10  Comparison  Between  Estimating  Requirements  and  Model 
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The  estimating  life  cycle  begins  with  the  detailed  system  design; 
that  is,  the  definition  of  the  allocated  system  functions.  The  phases 
overlap  in  time.  The  schedule  is  given. 

SLIM  ( Fi gure  1 2) .  SLIM  produces  a  primary  estimate  of  the  development 
cost  at  the  system  level.  It  provides  an  optional  "front-end"  estimate 
that  includes  the  Analysis  and  Design  phases.  The  Operations  and  Support 
cost  can  be  obtained  from  another  option.  Additional  options  provide 
estimates  for  computer  hours  and  documentation.  The  life  cycle  components 
are  described  as  overlapping  and  fixed  in  relative  size.  Milestone  events 
describe  the  beginnings  and  ends  of  the  phases. 

Tecolote  Research  (Figure  13).  A  single  system  level  estimate  is  proauced 
by  the  model.  No  allocation  of  effort  among  the  phases  is  given. 

Wolverton  (Figure  14).  A  very  detailed  matrix  allocates  the  system 
development  cost  into  seven  phases  each  composed  of  up  to  25  activities. 
Therefore,  a  complete  description  of  phases  and  activities  is  obtained  at 
the  system  level . 

Model  Compliance  with  Cost  Estimating  Situations.  In  Section  2,  the 
Air  Force  need  for  software  life  cycle  cost  data  was  described  in  terms 
of  five  cost  estimating  situations.  The  imprecise  nature  of  the  data 
requirements  and  the  dissimilarities  in  the  outputs  of  the  models  preclude 
the  creation  of  a  checklist  of  defined  data  items.  Therefore,  the  compari¬ 
son  of  the  model  outputs  with  the  needs  must  be  partly  subjective. 

The  comparison  of  model  outputs  with  estimating  needs  is  made  in 
two  dimensions;  scope  and  detail.  The  scope  of  the  estimating  needs 
describes  the  cost  elements  associated  with  the  life  cycle  phases.  The 
scope  of  a  model's  outputs  may  be  limited,  for  example,  to  the  Codinq  and 
Checkout  and  Test  and  Integration  Phases.  The  output  detail  describes  the 
extent  to  which  cost  elements  in  each  phase  represent  the  system  its  componen 
and  Its  associated  elements  such  as  Facilities  and  Training. 
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Figure  14  Comparison  Between  Estimating  Requirements  and 
Model  Outputs  -  Wolverton 
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Table  7  's  a  summary  comparison  of  the  model  outputs  with  the  needs 
described  in  Section  3.  A  liberal  interpretation  of  compliance  was 


exercised  in  each  case.  A  model  was  given  credit  in  terms  of  scope  if  it 
addressed  the  required  phases  regardless  of  the  possibility  of  differences 
in  definition  between  the  model  and  the  standard.  The  same  criterion  was 
used  to  describe  compliance  with  the  needed  level  of  detail. 

The  degree  of  model  output  compliance  with  each  stated  need  is 
described  using  a  scale  of  1  to  5,  where  5  indicates  nominal  compliance. 

In  general  a  model  was  given  one  point  for  including  each  of  the  five 
major  phases  in  its  estimates.  The  Installation  Phase  was  not  included 
in  the  scoring.  The  detail  scale  was  determined  less  objectively  ard 
depended  on  the  analyst's  view  of  the  extent  to  which  the  model  allows 
the  user  to  identify  the  different  system  elements  -  especially  as  they  were 
affected  by  phase  boundaries. 

Table  7  shows  that: 

•  The  models  are  better  able  to  provide  information  needed  in 
early  phases  of  the  life  cycle  than  in  the  later  ones. 

Most  of  the  models  provide  the  detail  needed  for  system  level  cost  estimates; 
but  none  of  them  rate  very  highly  when  the  CPCI  and  CPRC  levels  must  be 
described.  Therefore,  as  the  life  cycle  progresses  and  the  need  for 
estimates  becomes  directed  toward  the  components  of  the  system,  the  model 
ratings  decrease. 

The  best  performer  with  regard  to  detail  is  the  Wolverton  model.  Its 
matrix  of  phases  and  activities  executed  at  the  CPCI  level  is  able  to 
provide  most  of  the  detail  indicated  in  the  W8S.  It  was  not  rated  as  5 
because  the  model  structure  does  not  incorporate  a  mechanisin  for  accounting 
for  the  different  WBS  levels  and  their  associated  overheads.  It  was  also 
downgraded  because  it  does  not  identify  costs  for  facilities,  training  and 
installation. 
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TABLE  7  SUMMARY  OF  MODEL  COMPLIANCE  WITH  AIR  FORCE  ESTIMATING  REQUIREMENTS 
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Detail 
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Detail 
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1  5. 


FULL  SCALE  DEVELOPMENT 
Scope 
Detail 


2 

5 


4 

5 


4 

5 


1  1 
4  4 


4 

5 


4 

5 


2  4  4  1  1  3  5  4  4 

444555444 

344224544  j 

243224334! 

! 

! 

334224534j 
1  3  2  1  1  3  2  1  4! 

334334534 
1  3  2  1  1  3  2  1  4 


NOTE:  Numbers  indicate  degree  to  which  the  model  satisfies  the 
particular  estimating  requirements.  5  indicates  nominal 
satisfaction  of  the  requirement. 


The  Wolverton,  PRICE  S  and  Boeing  models  offer  more  detail  than  most 
of  the  other  models.  PRICE  5  allows  the  separate  identification  of  subsystem 
costs  and  includes  allocation  of  the  associated  integration  costs. 

The  most  common  failure  in  the  scope  dimension  wa'  the  omission  cf 
the  Operation  and  Support  Phase.  Only  the  SLIM  model  includes  an  estimate 
of  these  costs.  This  is  followed  in  frequency  by  omission  of  the  Analysis 
and  Design  Phases. 

As  the  system  develops,  the  scope  of  the  cost  information  naturally, 
becomes  less  because  the  initial  phases  are  completed.  That  is  >vny  t.ne 
models  are  rated  increasingly  higher  toward  the  bottom  of  Table  7. 

The  best  performer  in  the  scope  dimension  is  SLIM  followed  by  the  DoD 
Micro  Procedure,  PRICE  S  and  Wolverton.  These  latter  three  models  do  not 
include  the  O&S  Phase  in  their  estimates.  PRICE  S  is  downgraded  in  the 
first  two  estimating  situations  because  it  does  not  include  the  Analysis 
Phase  and  part  of  the  Design  Phase. 

From  Taole  7  ,ve  conclude  that: 

•  None  of  the  models  included  in  this  study  fully  satisfy  the 
Air  Force  need  for  information  with  regard  to  scope  or  detail. 

A  cotimon  fault  of  the  models  is  the  failure  to  properly  describe  WBS 
elements  that  cross  phase  boundaries.  For  example,  a  system  design  is 
often  changed  after  the  Coding  and  Checkout  Phase  begins  {Figure  15), 
but  few  models  or  data  sets  identify  the  design  effort  that  occurs  in  the 
Coding  and  Checkout  Phase.  The  PRICE  S  and  Wolverton  models  with  their 


ANALYSIS  PHASE 


aESIGN  PHASE 


CODING  AND  CHECKOUT 


CPCI  DEFINITION 


CPC  I  CODING 


Figure  15  Allocation  of  Work  Breakdown 

Structure  Elements  to  Life  Cycle  Phases 
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matrix  representations  of  the  cost  elements  do  permit  sucn  distinctions  to 
be  made.  However,  the  PRICE  S  terminology  does  not  allow  direct  comparisons 
with  the  Air  Force  phases  and  WBS.  In  general  we  can  state  that: 

•  The  models  tend  to  be  phase  oriented  and  do  not  properly 
describe  activities  that  cross  phase  boundaries.  This  pre¬ 
cludes  obtaining  data  compatible  with  both  management  plan¬ 
ning  (phase  related)  and  product  cost  (WBS). 

The  tested  models,  with  exception  of  PRICE  S,  tend  to  be  vague  about 
how  the  cost  of  developing  systems  relates  to  the  cost  of  developing  system 
components.  As  cost  analysts  well  know  a  system  is  more  than  the  sum  of 
its  parts.  Each  independently  developed  component  must  be  designed  and 
redesigned  in  concert  with  every  other  system  component.  Interface  and 
performance  specifications  must  be  analyzed  whenever  a  change  is  made  to 
any  part  of  the  system.  This  is  especially  true  of  large  software  systems, 
yet  software  cost  models  seldom  provide  explicit  descriptions  of  these  costs. 

PRICE  S  has  an  operating  mode  in  which  individual  subsystems  can  be 
estimated  using  the  normal  techniques  and  then  combined  along  with  a  speci¬ 
fied  integration  cost  to  produce  a  total  system  estimate.  Both  the  system 
and  subsystem  costs  are  presented. 

The  other  models  simply  add  subsystems  together  without  regard  to 
size  or  the  number  of  organizations  contributing  to  the  development.  It  is 
the  user's  responsibility  to  add  any  integration  costs  and  to  properly 
distribute  them. 

This  leads  to  our  observation: 

•  Although  most  of  the  models  use  the  summation  of  program, 
function,  or  module  sizes  to  make  their  cost  estimates,  only  one 
model  studied  provides  for  keepina  track  of  the  cost  on  a 
component  basis  and  accounts  for  the  cost  of  system  integration. 

None  of  the  models  provide  for  all  four  levels  of  system  definition 
called  for  in  the  Work  Breakdown  Structure. 
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5.2  MODEL  ESTIMATING  ACCURACY 

According  to  the  procedure  described  in  Section  4  and  using 
the  Relative  Root  Mean  Square  Error  measure  of  estimating  performance 
(Section  3.2),  each  of  the  subject  models  was  executed  using  as  many 
different  input  data  sets  as  possible  given  limitations  on  available  time 
and  historical  input  data. 

Appendix  C  shows  the  individual  results  obtained  by  executing  the 
models.  The  presentations  include  estimated  and  actual  values  of  the 
outputs  for  a  given  project  and  several  performance  measures.  The  follow¬ 
ing  paragraphs  present  the  analysis  of  the  estimating  pe*"formance .  The 
objective  is  to  identify  specific  attt'ibutes  of  the  model  structures  or 
the  data  sets  that  are  associated  with  estimating  accuracy. 

The  estimating  performance  of  the  subject  models  for  the  test  data 
sets  is  surmiarized  in  Table  8.  In  order  to  direct  attention  to  structures 
and  to  avoid  the  appearance  of  making -general  statements  about  the 
prediction  performance  of  the  specific  models,  the  models  are  identified 
by  codes  in  the  charts.  This  is  done  to  direct  attention  but  there  is 
no  intention  to  hide  the  results  obtained  for  each  model.  Therefore, 

Table  C-17,  in  Appendix  C,  shows  similar  information  as  Table  8  and 
includes  the  names  of  the  models. 

The  analysis  of  estimating  accuracy  was  accomplished  by  testing 
several  hypotheses  associated  with: 

•  Ltevelopment  environment  (Data  Set) 

•  Model  structure 

•  Model  calibration 

•  Use  of  system  size  as  a  ftiodel  input 


TABLE  8  SUMMARY  OF  MODEL  ESTIMATING  PERFORMANCE 


RMS  ERROR* 
MEAN  PROJECT  SIZE 
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Four  of  the  models  (A,  E,  H,  J)  were  executed  on  all  three  data 
sets.  The  measures  of  accuracy  obtained  from  these  12  cases  were 
subjected  to  a  two-way  analysis  of  variance  [45].  The  analyis  of 
variance  is  a  systematic  way  of  inferring  the  statistical  significance 
of  the  relative  contributions  of  each  model  and  data  set  to  the  total 
sample  error*.  The  contribution  of  each  model  to  the  total  Root  Sum 
Square  Error  is  observed  while  controlling  the  contributions  made  by  the 
different  data  sets.  The  contributions  of  the  data  sets  can  be  similarly 
analyzed.  This  procedure  was  used  to  test  the  first  three  hypotheses 
above. 


The  12  test  cases  are  tabulated  separately  in  Table  9.  Included 
In  the  comparisons  are  two  regression  models  (A  is  not  calibrated,  E  is 
calibrated),  a  heuristic  model,  H,  and  the  phenomenological  model,  J. 

The  heuristic  and  phenomenological  models  have  calibration  modes  that 
can  be  used  before  making  estimates. 

The  two-way  analysis  of  variance  produces  an  inter-column  f  statistic 
of  1.97  and  an  inter-row  F  statistic  of  1.95.  The  null  hypotheses  that  the 
row  and  column  effects  on  the  total  mean  square  error  are  zero,  can  both 
be  rejected  with  an  80  percent  level  of  confidence.  This  means  that  taken 
as  groups  there  are  differences  in  estimating  accuracy  among  the  models 
and  among  the  environments  represented  by  the  three  test  data  sets. 


*  Citing  any  statistical  procedure  implies  certain  assumptions  about  the 
characteristics  of  the  populations  represented  by  the  test  subjects. 

The  analysis  of  variance  is  restricted  to  normal  populations  with  equal 
variances.  There  is  an  assumption  of  linearity  of  the  contributions 
of  group  differences  to  the  total  sample  difference,  independence  and 
others.  We  cannot  be  certain  that  these  conditions  are  even  partially 
satisfied.  Therefore,  the  presentation  of  the  statistical  results  ts 
not  made  as  proof,  but  only  as  an  indication  of  possible  support.  0^ 
course,  even  under  rigorous  satisfaction  of  all  conditions,  causality 
is  never  proved.  Other  observations  will  be  offered  in  this  secticr 
without  statistical  justification,  but  only  justified  by  conformin^, 
to  intuitively  acceptable  patterns. 
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Comparisons  among  the  individual  row  and  column  members  were  made 
to  learn  how  the  different  models  and  data  sets  contributed  to  the  over¬ 
all  results. 

Table  10  is  a  presentation  of  the  error  totals  by  rows  and  columns. 

The  table  entries  are  the  differences  between  the  marginal  values.  For 
example,  the  first  row  and  first  column  entry  in  the  model  table,  2.13, 
is  the  difference  between  the  total  error  on  all  three  data  sets  for 
Model  A  (4.01)  and  the  similar  measure  for  Model  E  (1.88).  Th^  null 
hypothesis  that  the  difference  is  zero  can  be  rejected  with  an  80  percent 
level  of  confidence  for  values  in  the  model  table  greater  than  1.73  and 
in  the  data  set  table  greater  than  2.00.  Asterisks  indicate  the  signifi¬ 
cant  values. 

Table  10  indicates  that  significaf!t  differences  in  estimating  accuracy 
exist  between  Model  A  and  each  of  the  other  models.  However,  the  accuracy 
differences  among  the  other  models  are  not  significant.  Mooel  A  (Aerospace) 
is  not  recalibrated  to  any  of  the  test  data  sets;  and  Model  E  has  the  same 
form  as  A  (MM  =  al*^)  but  has  been  recalibrated.  Therefore,  the  results 
indicate  the  effect  that  recalibration  has  on  estimating  performance. 

The  recalibration  of  the  form  used  by  Model  A  produces  a  model  that 
has  the  same  estimating  performance  as  the  other  two  models.  This  suggests 
that; 

•  Recalibration  is  the  primary  factor  contr’buting  to  the  difference 
in  estimating  performance  experienced  by  the  models  in  Tat'e  9 
and 

•  The  contribution  to  estimating  accuracy  related  to  model  struc¬ 
ture  is  not  significant  when  each  of  the  model  structures  have 
been  calibrated  to  a  given  data  set. 
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TABLE  10 


^^IPWISE  COMPARISONS  - — •'  ~ING  PERFORMANCE 
Table  Entries  are  Differences  in  total  RMS  Error 


COMPARISONS  OF  MODELS 


■'‘otal  RMS 

A  E  H  J 

4.01  1.88  2.12  1.33 

A  4.01 

E  1.88 

!  H  2.12 

'  J  1.33 

2.13* 

1.89*  0.24 

2.68*  0.55  0.79 

COMPARISONS  OF  DATA  SETS 


Total  RMS 

Data  Set  Error 

Comm.  DSDC  SEL 

2.62  4.70  2.08 

Comm.  2.62 

DSDC  4.70 

SEL  2.08 

2.08* 

0.54  2.62* 

*  Difference  is  significant  with  at  least  an  80  percent  level  of  confidence 
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The  second  statement  derives  from  the  fact  that  the  three  calibrated 
model  structures  are  very  different.  E  is  the  simple  form  MM  =  al^,  H 
is  the  PRICE  S  model  and  J  is  SLIM. 

The  pairwise  comparisons  of  the  estimating  performance  of  the  models 
among  the  different  data  sets  indicates  that  the  accuracy  measurements  are 
significantly  different  for  two  data  sets.  This  indicates  that: 

•  The  development  environment  is  a  significant  factor  affecting 
the  relative  estimating  performances  of  the  models  tested. 

A  given  model  structure  will  perform  better  in  some  environments  than 
others.  This  finding  indicates  the  necessity  of  learning  the  specific 
attributes  of  a  development  environment  that  determine  when  one  or  another 
model  structure  should  be  used.  This  view  is  further  supported  by  the 
performance  of  Model  J,  SLIM,  shown  in  Table  9.  Model  J  is  the  best  per¬ 
former  for  the  first  two  data  sets  and  the  worst  performer  on  the  third.  It 
is  essential  that  we  identify  the  characteristics  of  the  model  structures 
relative  to  those  of  the  development  environment  that  affect  the  ability 
of  the  model  to  make  accurate  predictions.  This  result  also  substantiates 
the  need  for  making  accuracy  evaluations  on  as  broad  a  range  of  environments 
as  possible.  In  effect  we  are  saying: 

9  The  effect  of  development  environment  on  estimating  performance 
precludes  the  possibility  of  obtaining  generally  applicable 
measures  of  the  performance  of  any  model  or  model  structure 
without  applying  additional  controls. 

The  measurement  of  the  effect  of  recalibration  on  estimating  per¬ 
formance  was  repeated  using  some  of  the  other  results  in  Table  8.  Since 
some  of  the  models  are  recalibrated  on  each  data  set  and  others  are  not, 
the  estimating  accuracy  of  one  set,  the  calibrated  models,  was  compared 
with  the  non-calibrated  ones.  In  order  to  make  the  comparisons  on  the 
same  basis  only  one  data  set  was  used  and  the  models  were  separated  by  type. 
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The  four  non-cal ibrated  Regression  type  models  (A,  B,  C,  D)  with 
a  relative  RMSE  of  6.10  were  found  to  be  not  significantly  different  from 
the  0,933  value  obtained  for  the  calibrated  Regression  models  (E).  This 
is  because  the  large  range  in  the  error  values  has  a  correspondingly  large 
variance  which  reduces  the  statistical  significance  of  differences  between 
the  mean  and  any  given  value.  This  is  true  even  when  Model  C  is  eliminated 
from  the  group. 

The  exercise  was  repeated  for  the  Heuristic  model  types.  In  this 
case  the  calibrated  models  include  F,  G,  and  I.  These  with  their  relative 
RMSE  of  0.991  were  compared  with  the  1.44  value  obtained  with  Model  H. 

The  difference  is  not  significant. 

These  two  experiments  conducted  on  a  single  data  set  failed  to  sub¬ 
stantiate  the  findings  previously  obtained  by  executing  four  models  using 
all  three  data  sets.  However,  there  is  an  important  difference  in  the 
two  investigations.  In  the  first  analysis  the  effect  of  calibration  was 
obtained  by  using  the  same  model  structure  to  represent  both  non-cal ibrated 
and  calibrated  models.  In  the  second  analysis  a  group  of  structures  was 
usea  to  represent  the  non-cal ibrated  models.  The  results  may  only  indicate 
that  there  is  too  much  variation  among  the  models  to  allow  a  significant 
comparison  of  estimating  performance.  This  also  indicates  that  better 
criteria  for  stratifying  the  model  structures  are  needed.  This  view  is 
also  supported  by  the  fact  that  the  first  analysis  was  more  limited  in  the 
number  of  model  structures  it  included.  Considering  the  effect  of  cali¬ 
bration  on  estimating  performance  it  is  necessary  to  know  the  portion  of 
the  variation  in  performance  among  the  model  structures  that  would  be 
eliminated  by  calibration.  Only  after  the  model  structures  are  recalibrated 
would  it  be  possible  to  explore  the  effects  on  accuracy  of  such  model 
attributes  as:  number  and  types  of  input  variables,  level  of  the  initial 
estimate,  the  method  of  making  the  initial  estimate  and  the  method  for 
making  subsequent  estimates. 


Examination  of  the  results  in  Table  8  suggests  some  additional 
findings  that  are  presented  without  statistical  justification.  Considering 
the  previously  observed  effects  on  performance  related  to  recalibration 
and  environment,  these  inferences  must  be  considered  tentative. 

The  only  model  in  the  test  group  that  does  not  use  a  form  of  code 
size  as  an  input  (G,  DoD  Micro  Procedure)  has  estimating  performance  that 
is  comparable  to  the  other  models  of  its  type.  This  suggests  that: 

•  Models  that  do  not  use  size  as  an  input  may  perform  as  well  as 
those  that  do. 

If  this  is  true  there  may  be  an  increase  in  accuracy  obtained  by 
using  a  non-size  input  in  the  early  phases  of  the  life  cycle  when  size 
IS  known  less  precisely  (see  Section  7.1). 


TABLE  11 

AVERAGE  ESTIMATING  PERFORMANCE 
Average  Relative  Root  Mean  Square  Error 


All  Models  and  Data  Sets 

1.930 

Non-Cali  bra ted  Models 

3.260 

Calibrated  Models 

0.592 

Table  11  indicates  that: 

•  The  average  RMS  Error  for  all  tested  models  is  unacceptably 
large  for  Air  Force  estimating  purposes, 

•  The  use  of  models  that  are  not  calibrated  to  a  given  development 
environment  can  lead  to  very  large  estimating  errors, 

•  The  best  performance  obtained  by  any  group  of  the  models  tested 
(calibrated,  RRMSE=0.592)  is  not  adequate  for  Air  Force  needs. 
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When  using  the  Relative  RMS  Error  to  describe  estimating  accuracy, 
It  should  be  understood  that  the  values  do  not  represent  expected  values 
of  error  for  estimating  situations.  The  RMS  Error  is  a  weighted  measure 
that  penalizes  large  deviations  from  predicted  values.  Appendix  3  shows 
that  large  deviations  are  common  among  the  models  and  justifies  the  use 
of  this  statistic. 
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6  ANALYSIS  OF  RESULTS 

6.1  ENVIRONMENT 

A  model  is  an  abstraction  of  some  real  process  that  we  try  to 
represent  using  some  selected  variables  and  a  hypothesized  relationship 
among  them.  When  we  construct  the  model  we  hope  that  we  have  found  the 
important  variables.  That  is,  that  the  forces  at  play  that  determine  the 
outcome  of  the  process  are  adequately  represented  by  the  model .  We  never 
know  for  sure  if  this  is  true;  that  is,  if  the  model  is  valid  for  all 
circumstances.  We  can  only  state  that  if  it  is  observed  for  a  long  time  over 
a  large  part  of  the  input  domain  and  if  its  behavior  is  consistent  with  exper¬ 
ience,  we  begin  to  feel  comfortable  that  it  will  always  behave  as  the  proto¬ 
type  behaves.  Then,  the  model  is  believed  to  be  a  valid  representation  of 
the  system  behavior. 

However,  we  never  know  when  we  may  wander  away  from  the  domain  on 
which  the  model  is  valid.  The  Ptolemaic  model  of  the  solar  system  was 
believed  valid  until  deviations  between  the  model  and  observation  were 
established.  This  process  took  hundreds  of  years. 

The  Newtonian  model  of  dynamic  behavior  was  believed  valid  until 
the  relativistic  domain  was  encountered.  In  both  cases  behavior  was  in 
concert  with  theory  only  until  phenomena  which  had  always  been  present 
became  evident.  Then  the  models  were  understood  to  be  either  special 
cases  of  a  more  general  representation  or  simply  invalid. 

Models  of  software  development  are  in  a  primitive  state.  It  is 
not  possible  to  completely  explain  observed  phenomena  in  controlled  environ¬ 
ments  let  alone  make  general  statements.  Therefore,  we  must  view  these 
findings  as  indications  of  possible  relationships  and  nothing  more. 

The  forces  acting  on  a  software  development  project  are  many,  com¬ 
plex,  often  subtle  aid  even  counter-intuitive  [8][46].  They  reflec* 
human  performance  and  its  variability  among  individuals.  Individual 
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performance  itself  varies  according  to  many  circumstanceb .  The  forces  also 
reflect  group  behavior  and  the  availability  of  resources.  The  total  of  all 
factors  affecting  performance  constitute  the  development  environment.  If 
the  personnel,  equipment  and  management  structure  in  a  given  organization 
remain  relatively  unchanged  over  a  period  of  time,  we  have  a  better  prob¬ 
ability  of  obtaining  valid  models  of  the  organizational  behavior.  This  is 
tacjiuse  many  of  the  forces  that  may  have  profound  effects  on  performance 
are  not  changing  and  therefore  need  not  be  considered  explicitly.  Their 
t'fects  will  be  included  ifi  the  oarameters  of  the  model.  However,  if  the 
orga.iizational  environment  is  changed,  the  model  may  become  invalid  and 
its  behavior  may  depart  radically  from  its  prototype. 

'he  point  of  this  discussion  is  that  the  results  obtaineu  nerc- 
we>"e  observed  for  three  different  organizations  that  constitute  three 
different  development  environments.  We  have  observed  that  the  collective 
behavior  of  the  model  types  differ  among  the  environments.  This  suggests 
we  should  be  very  cautious  about  drawing  inferences  about  the  behavior  of 
any  model  in  the  evaluation  and  apply  them  to  another  environment  such  as 
our  own  organization.  Notice  that  the  rankings  of  the  models  on  the  basis 
of  accuracy  was  not  consistent  for  the  three  data  sets.  Some  of  the 
models  moved  from  best  to  worst  depending  on  the  test  data.  We  have 
observed  some  behavior  but  we  must  be  very  cautious  about  making  any 
generalizations.  We  have  the  basis  for  making  some  hypotheses,  but  we 
haven't  proved  anything.  What  has  been  demonstrated  in  this  study  may 
be  the  result  of  the  choice  of  environments  and  project  types. 

6.^'  THE  EFFECTS  OF  INPUT  ESTIMATING  ERRORS 

Section  3,  Evaluation  Procedure,  indicates  that  missing  'np-,t 
data  were  est’mated  when  necessary  to  execute  the  models.  It  is  desirable 
to  test  the  effect  that  such  estimates  may  have  on  the  measurement  of 
'cmparative  estimating  accuracy. 
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The  model  estimates  are  subject  to  random  errors  associated  with 
the  values  of  theinnuts.  This  means  that  the  observed  differences  iri 
estimating  performance  are  expected  values  and  they  are  significant  subject 
to  some  uncertainty.  It  is  possible  that  in  some  cases  the  uncertainty 
may  be  great  enough  to  prevent  acceptance  of  the  result.  It  was  not 
possible  to  establish  measures  of  variation  for  all  the  model  estimates 
for  all  the  data  sets.  This  should  be  done.  It  was  possible  to  do  it  for 
two  models  and  the  effect  on  the  comparative  accuracy  measurement  is  pre¬ 
sented  here.  However,  the  presentation  is  more  an  illustration  of  how  the 
effects  of  errors  in  the  input  values  should  be  included  in  the  analysis  of 
the  model  estimating  accuracy  rather  than  a  representative  finding. 

Two  models,  F  and  G,  were  selected  to  test  sensitivity  to  input 
errors.  These  two  models  use  different  inputs;  and  each  has  several  esti¬ 
mated  inputs.  As  shown  in  Table  8  ,  Summary  of  Model  Estimating  Performance, 
models  F  and  G  have  relative  RMS  errors  of  0.787  and  1.26  respectively.  We 
want  to  determine  whether  the  variations  in  the  predictions  produced  by  the 
uncertainties  in  the  input  values  allows  us  to  accept  the  difference  with 
reasonable  confidence. 

Tables  12  andl3,  taken  from  Appendix  D,  present  the  inputs  for  the 
two  models.  Model  F  has  three  estimated  inputs  and  Model  G  has  five. 

The  tables  contain  the  estimated  error  associated  with  each  input. 

Each  model  was  executed  100  times  using  input  values  selected  at 
r'diiauni  Trom  normal  distributions  defined  by  the  given  means  and  standard 
deviations.  The  distributions  of  the  sampled  values  were  truncated  to 
aomit  only  positive  values. 

The  executions  produced  the  following  results: 


6-3 


TABLE  12.  INPUTS  FOR  MODEL  F,  BOEING  COMPUTER  SERVICES 


AiliN3  ViVa/3a03  3NI1-NO 
nooi  90830  ON 
90830  3Nn-N0 
SWd03-083VM 
3003 -083 VW 
lOH 
•0*j 
MS  •iSIX3 


TABLE  13.  INPUTS  FOR  MODEL  G,  MICRO  ESTIMATING  PROCEDURE 


(X 


<n 

CVJ 

o> 

VO 

• 

o 

CO 

9> 

QO 

VO 

VO 

GO 

VO 

1 


MODEL 


Relative  RMS  Error 

1  •>,/ 

F 

1 

G 

Mean 

0.787 

1 .26 

j 

Standard  Deviation 

0.00893 

0.0458  i 

Stand.  Dev.  of  Mean 

0.00282 

0.0102 

1 

The  hypothesis  that  the  actual  difference  between  the  two  means  is  zero 
:an  be  rejected  with  a  confidence  level  greater  than  .999.  This  finding 
indicates  that  the  relative  estimating  performance  between  the 
two  selected  models  is  not  a  random  result  arising  from  uncertainties  in 
the  model  i nputs . 

6.3  MODEu  CALIBRATION 

.Ne  nave  snown  chat  tne  calibration  of  riodel  paraineters  may  oe  as 
■•"portant  as  model  structure  in  explaining  estimating  accuracy,  "wc  of 
tne  models  tested  incorporate  calibration  into  the  estimating  process. 

That  is,  these  models  require  or  suggest  that  data  representative  of  the 
aevelopment  environment  be  used  to  compute  model  parameters  before  using 
the  mooel  to  estimate  new  projects.  In  effect  the  creators  of  these 
models  are  saying  that  in  addition  to  certain  fixed  parameters  that  are 
permanent  parts  of  the  modei  structure,  there  are  additional  values  that 
describe  the  effects  of  the  circumstances  under  which  a  project  is  executec. 
These  values  are  ideally  constant  in  a  stable  environment,  but  some  users 
apply  judgement  based  on  experience  to  the  selection  of  values  for  a  given 
estimating  situation. 

Mooel  H,  PRICE  S,  utilizes  two  calibration  parameters,  one  sensi*‘ive  :  ■ 
to  total  cost,  the  other  to  development  time  [47],  The  user  of  the  model  is 
instructed  to  execute  the  calibration  mode  of  the  model  to  obtain  values 
of  tne  two  parameters  from  past  projects  that  are  representative  of  the 
expected  development  environment. 
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Model  J,  SLIM,  has  one  calibration  parameter  which  is  said  to 
represent  the  efficienty  with  which  a  given  organization  can  produce  a 
given  type  of  system  [48 J.  This  model  also  has  a  calibration  mode  for 
obtaining  the  parameter  from  project  experience. 

The  calibration  of  each  of  the  models  did  not  produce  constant 
values  of  the  paraireters.  It  could  be  argued  that  the  reasons  the  para¬ 
meters  varied  was  that  the  environments  were  not  similar.  But  this  is 
circular  reasoning.  Since  in  both  instances  there  is  no  way  to  measure 
the  calibration  parameters  directly,  we  can  only  observe  their  effects  on 
the  model  estimates.  If  deviations  in  the  calibration  parameters  are 
proof  that  development  environments  are  different  from  one  another  when 
it  was  assumed  they  were  representative  of  the  development  environment, 
then  we  must  admit  that  we  don't  know  what  constitutes  the  a  priori 
indicators  of  environment.  If  that  is  ti ue,  we  have  no  way  to  know  if 
any  new  project  will  be  represented  by  one  past  environment  or  another 
and  will  not  know  which  parameters  to  use  for  estimating. 

It  is  necessary  in  any  calibration  of  model  parameters  to  know  when 
calibration  is  necessary  and  when  any  given  parameters  are  applicable  in  an 
estimating  stiuation.  In  the  case  of  models  H  and  J  the  calibration  con¬ 
stants  vary  significantly  among  projects  within  a  given  organization.  This 
coupled  with  the  sensitivity  of  the  model  outputs  to  changes  in  the  cali¬ 
bration  constants  makes  the  solution  of  the  calibration  problem  the  key  to  suc¬ 
cessful  model  performance. 

Calibration  of  models  H  and  J  was  initially  accomplished  by 
randomly  selecting  several  projects  from  a  set  of  test  projects  and 
executing  the  calibration  modes  of  the  models.  Significant  variation  in 
values  of  the  calibration  parameters  were  obtained  for  both  models.  Since 
the  number  of  projects  available  fo  measuring  estimating  accuracy  were 
limited,  it  was  decided  to  systematically  vary  the  calibration  variables 
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to  obtain  the  best  estimating  performance.  This  biases  the  evaluation  and 
produces  the  best  possible  performance  fo*”  a  given  data  set,  but  we  did  not 
want  to  compromise  either  model's  performance  because  of  a  chance  selection  of 
projects  for  calibration.  Observing  the  effects  of  changes  in  the  calibra¬ 
tion  parameters  on  the  model  predictions  simulates  the  learning  process 
that  would  occur  as  experience  is  obtained  using  a  model  in  the  same  environ¬ 
ment.  In  the  case  of  each  model  the  experience  mode  gave  better  estimates 
than  the  model  calibration  mode. 

The  recalibration,  E,  of  the  Aerospace  model  (A)  was  accompMshec  by  a 
linear  least  squares  fit  of  the  logarithm  of  the  program  size  in  lines  of 
source  code  for  the  three  data  sets.  Since  the  parameters  obtaineo  \ere 
used  to  show  estimating  performance  on  the  same  data  sets,  the  recal ipra' :cn 
results  are  comparable  to  those  obtained  using  the  extended  cal ibr atirn 
procedure  cn  models  H  and  J. 

When  model  structures  are  calibrated  to  a  given  development  environ¬ 
ment,  the  effect  of  structural  differences  tend  to  disappear.  This  is  true 
at  least  for  an  accuracy  measurement  derived  from  a  total  effort  estimate. 
However,  this  ability  to  predict  is  very  sensitive  to  the  environment  and 
it  is  not  known  how  the  success  of  a  model  in  one  environment  is  rplated 
to  success  in  another.  We  don't  know  how  to  measure  an  environment's  attri¬ 
butes  to  know  if  it  belongs  to  the  same  population  represented  in  a  given 
data  set  that  was  used  to  obtain  the  model  parameters.  We  only  know  that 
for  the  case  where  it  is  known  that  a  project  is  a  member  of  a  cata  set, 
the  performance  is  within  observed  limits.  But  even  within  a  calibration 
data  set  the  differences  in  estimating  accuracy  can  be  large.  'We  have 
shown  that  calibration  is  effective  I'l  increasing  estimatinv  accuracy,  jt 
we  don't  know  when  calibration  is  necessary  or  which  nistorical  projeoci 
to  include  in  a  calibration  to  obtain  the  best  accuracy.  We  have  seen  some 
classifications  of  development  projects  according  to  such  descripricns 
commercial,  scientific,  and  time  critical  ''uCl  :23]  .'4b'.  but  t.:ese 
definitions  are  never  explained. 
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The  problem  of  selecting  a  model  for  a  given  estimating  situation 
may  be  stated  more  clearly  in  terms  of  specific  models.  For  example,  assume 
we  n.jst  select  among  models  E,  H,  and  J  to  make  an  estimate  of  software 
cost.  This  presumes  that  data  are  available  to  calibrate  the  models.  First 
we  must  decide  if  the  development  environment  is  comparable  to  one  described 
by  one  of  the  three  test  data  sets.  If  the  development  environment  is 
similar  to  SEL,  Models  E  or  H  should  provide  the  most  accurate  estimate. 

If  the  environment  is  more  like  the  Commercial  or  DSDC  environments,  Model 
J  should  be  the  most  accurate.  The  problem  is  that  we  don't  know  which  of 
the  many  attributes  that  may  describe  a  development  environment  are  suffi¬ 
cient  to  determine  the  equivalence  of  two  environments  for  estimating 
purposes. 

6.4  THE  USE  OF  UNMEASURABLE  VARIABLES  AND  PARAMETERS 

It  is  possible  to  identify  two  points  of  view  that  are  evident  in 
the  choices  of  model  predictor  variables  and  parameters.  One  group  of 
models  includes  only  variables  that  can  be  readily  measured  at  some  time 
during  or  after  the  development  of  a  software  system.  Definitions  may 
differ  in  detail,  (e.g.  program  size)  but  they  can  be  measured.  The 
important  consideration  is  that  the  estimate  of  the  variable  can  be 
verified  subsequently  and  if  the  prediction  is  off,  we  will  know  whether 
it  is  because  the  estimate  or  the  model  was  wrong. 

The  second  type  of  model  includes  subjective  variables  that  may  be 
representive  of  important  attributes  of  software  development  but  are  not 
expressable  as  measurable  quantities.  An  example  would  be  a  variaole  repre¬ 
senting  the  difficulty  or  complexity  of  the  development  effort  relative  to  tne 
ability  of  the  development  group.  It  may  be  possible  to  obtain  a  consensus 
among  analysts  about  what  these  values  should  be,  but  the  values  are  never 
observable  before  or  after  the  software  development.  Therefore,  if  the 
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estimate  subsequently  differs  from  experience,  we  cannot  know  if  the  dif¬ 
ference  was  because  the  predictor  was  not  estimated  properly  or  the  model 
failed  to  perform.  We  can  discuss  whether  the  initial  extimates  were  appro¬ 
priate  given  past  experience  and  the  characteristics  of  the  software  and 
the  development  environment,  but  the  results  are  always  subject  to  inter¬ 
pretation.  This  is  especially  true  if  the  model  estimates  are  sensitive 
to  the  subjective  parameters. 

The  exclusive  use  of  measurable  variables  may  unnecessarily  limit 
:rie  development  of  model  structures.  It  may  be  that  it  is  not  possible, 
given  the  state  of  the  software  engineering  art,  to  identify  and  quantify 
the  variables  that  determine  the  cost  of  developing  software.  On  the  otner 
hand,  too  much  reliance  on  subjective  inputs  may  perpetuate  the  concept  of 
estimating  as  a  black  art  that  cannot  be  explored  objectively.  It  nay  te 
best  at  this  time  to  pursue  a  policy  that  favors  the  maximum  use  of  measur¬ 
able  predictors  but  recognizes  the  possibility  of  using  subjective  inputs. 
Subjective  estimators  supported  by  carefully  documented  guidelines  for  their 
yaluatation  may  provide  reliable  estimates.  Therefore,  we  should  accept  such 
models  as  legitimate  interim  steps  which  may  provide  accurate  estimates 
and  possibly  insights  into  more  objective  measures. 

The  problem  remains,  however,  of  making  an  objective  measurement 
of  the  prediction  accuracy  of  models  that  include  unmeasurable  inputs. 

There  is  a  tendency  to  play  with  the  subjective  models  until  good  results 
are  obtained.  The  fact  that  good  results  can  be  obtained  may  be  signifi¬ 
cant  by  itself,  but  model  sensitivity  may  ensure  this.  The  process  may  be 
compared  to  the  natural  use  of  the  model  in  any  stable  prediction  environ¬ 
ment.  But  the  fact  remains  that  the  comparisons  between  the  models  are 
not  being  made  on  the  same  basis.  Our  conclusion  is  that  the  only  evalua¬ 
tion  of  a  subjective  model  (one  that  Includes  one  or  more  major  parameters 
that  can't  be  measured)  is  to  observe,  if  in  a  stable  environment  where 
the  type  of  software  being  developed  is  the  same  and  there  are  no 
upheavals  in  the  organization  O”  the  personnel,  whether  the  parameters 
behave  in  a  predictable  fashion.  If  this  is  true,  it  would  seem  that  the 


model  can  be  used  in  that  environment  for  the  tested  type  of  software. 

But  if  given  a  stable  environment,  the  parameters  that  give  the  best  a 
posteori  predictions  do  not  behave  in  a  moderate,  predictable  fashion, 
then  the  model  probably  is  not  a  useful  tool  for  that  environment. 

In  either  case  no  general  conclusions  may  be  drawn  about  the  values 
of  the  models  in  other  environments  or  for  that  matter  in  the  same  environ¬ 
ment  if  the  type  of  development  changes  significantly.  Here  significant 
is  not  definable  because  there  are  no  reliable  principles  that  define  the 
domain  of  the  predictors.  Therefore,  we  never  know  when  they  may  become 
unreliable  or  invalid.  This  will  be  true  to  some  extent  for  the  objective 
models,  but  it  is  of  much  greater  import  for  the  subjective  models. 

Our  evaluation  objective  is  to  learn  if  certain  model  structures 
can  be  demonstrated  to  be  better  predictors  than  others  and  if  so  to 
recommend  how  future  model  development  should  proceed.  Our  interest  in 
the  subjective  models  should  they  exhibit  positive  qualities  is  perhaps 
to  learn  if  the  subjective  variables  can  be  quantified  or  at  least  to 
learn  the  boundaries  of  the  regions  within  which  they  may  be  used  success¬ 
fully. 


6.5  APPLICABILITY  OF  THE  EVALUATION 

The  evaluation  is  made  using  data  obtained  from  three  different 
environments.  They  were  obtained  using  questionnaires  to  supplement 
other  measures  (e.g.  time  reports).  The  system  developers  were  contacted 
whenever  data  seemed  inconsistent  with  itself  or  failed  lO  satisfy  the 
analyst's  intuition.  However,  as  was  the  case  with  the  SDC  data  and  as 
Nelson  [50]  expressed  so  well: 

"All  the  data  used  from  both  the  statistical  analysis 
and  the  literature  were  data  of  opportunity,  i.e.,  we 
took  what  we  were  able  to  get  in  the  time  available. 

Hard  data  on  the  costs  of  computer  programming  .  .  . 
are  scarce  commodities  both  in  computer  programming 
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organizations  and  in  the  published  literature.  Few 
numerical  data  are  recorded;  fewer  yet  are  recorded 
under  'controlled'  conditions,  and  still  fewer  are 
suitacle  for  generalization  to  other  situations  .  .  . 

The  responoents  to  the  questionnaire  were  under  no 
obligation  to  assure  completeness  and  accuracy  even 
when  data  were  readily  available.  Because  they  were 
suspect,  some  of  the  data  collected  were  rejected 
prior  to  the  analysis.  But  even  those  data  used  in 
the  analysis  are  likely  to  have  a  variation  in 
reliability  ..." 

This  is  a  frank  and  somewhat  negative  evaluation  of  the  data  quality, 
but  before  anyone  hastens  to  disparage  the  results  because  they  don't  support 
their  own  experience  or  because  they  fail  ?o  show  their  favorite  model  to 
be  as  good  a  performer  as  they  believe  it  to  be,  we  should  state  that  the 
data  used  were  of  as  good  quality  as  any  other  available  for  cost  analysis 
and  considerably  better  than  most.  Given  the  opportunity  to  analyze  most 
software  development  data  in  detail  [51]  [52]  we  would  find 
the  same  kinds  of  deficiencies  that  we  have  expressed  in  assessing  the 
data  on  which  this  analysis  is  based.  If  someone  has  executed  a  model 
that  has  historically  performed  better  for  them  than  is  presented  here, 
then  we  would  argue  that  there  is  a  fortuitous  fit  between  the  character¬ 
istics  of  the  model  and  the  environment  in  which  it  is  being  used.  Nothing 
in  this  report  should  be  constr-.ed  as  a  general  statement  about  model  per¬ 
formance.  We  can  only  describe  the  models'  performances  as  they  were  repre¬ 
sented  in  a  very  careful  analysis  that  used  good  quality  data  and  an 
objective  comparison. 

This  evaluation  has  shown  what  other  researchers  have  already  exoressed 
[31]  [43]  [47]  [48]  [49]  [50]  [53]  [54'  [55];  that  model  performance  given 
the  limitations  on  standard  development  ^:''Cv.eaures ,  oenmtions  ana  under¬ 
standing  of  the  cost  driving  factors  is  very  much  environment  dependent. 
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We  think  the  results  give  a  good  indication  of  what  can  be  expec¬ 
ted  when  these  models  would  be  used  by  an  outside  agancy  such  as  the  Air 
Force  when  trying  to  make  estimates  in  support  of  the  Major  Weapon  System 
Acquisition  Process.  This  was  our  objective  and  we  think  we  accomplished 
it.  We  readily  concede  that  given  the  opportunity  to  calibrate  a  given 
model  over  a  period  of  time  in  a  stable  environment,  that  better  performance 
might  be  obtained  than  is  presented  here,  but  that  is  a  different  estimating 
situation  from  that  considered  in  this  analysis. 


7  RECOMMENDATIONS 

Four  of  the  five  cost  estimating  situations  described 
in  Section  2  occur  when  very  little  is  known  about  the  development 
environment.  Given  the  si'jnificance  of  the  effects  of  environment  and 
calibration,  it  is  necessary  to  develop  methods  whereby  the  Air  Force 
can  overcome  the  disadvantage  of  operating  at  a  distance.  The  Air  Force 
must  identify  and  obtain  the  data  items  that  will  ensure  accurate  estim¬ 
ates  in  any  specific  environment.  These  would  include  data  that  char¬ 
acterize  the  past  performance  in  that  environment  as  well  as  items  of 
importance  to  the  project  being  estimated.  The  Air  Force  must  obtain 
the  visibility  into  a  cost  estimating  situation  that  is  presently  avail¬ 
able  only  to  the  persons  who  are  members  of  the  organization  and  who  have 
first  hand  experience  with  that  organization’s  performance. 

Under  this  approach  model  structures  will  be  sought  that  are  easily 
calibrated  to  a  given  organization  using  auditable  historical  data  provided 
by  the  organization.  These  data  and  others  specific  to  a  given  project 
would  enable  the  Air  Force  tu  validate  a  proposal  for  software  development. 

A  collection  of  such  data  would  characterize  any  group  of  organizations 
and  would  be  used  in  the  initial  phases  of  the  life  cycle. 

The  following  recommendati ons  describe  a  course  of  action  that  wi""! 
provide  the  above  capability.  The  objective  of  the  recommendations  is  to 
place  the  analysis  and  synthesis  of  Air  Force  software  cost  estimating 
models  on  a  systematic  basis. 

7.1  MODEL  DEVELOPMENT 

The  results  of  the  accuracy  evaluation  suggest  that  the  best  way 
to  make  cost  estimates  is  to  use  the  simplest  model  structure  and  to 
calibrate  its  parameters  to  represent  the  development  environment.  However, 
this  approach  fails  to  consider  several  factors; 

•  The  measurement  of  estimating  accuracy  in  the  present 
study  does  not  consider  the  need  to  estimate  the  elements 
of  the  Work  Breakdown  Structure, 


7-1 


•  None  of  the  model  structures  achieved  the  needed  level  of 
accuracy, 

•  The  accuracy  of  the  models  in  this  analysis  is  overstated  because 
it  reflects  no  error  attributable  to  uncertainty  in  the  input 
values.  This  error  changes  during  the  life  cycle  and  is 
different  to  each  input. 

It  is  very  likely  that  the  incorporation  of  these  considerations 
into  a  further  analysis  would  affect  the  findings  regarding  the  model 
structure.  Whereas  a  simple  structure  may  be  adequate  for  estimating  total 
development  effort,  a  more  complex  structure  is  needed  to  define  cost 
elements  for  a  single  phase  at  a  lower  level  of  the  WBS.  Therefore,  the 
present  findings  must  be  considered  inconclusive  regarding  the  effects  of 
particular  structures  on  prediction  accuracy.  The  effects  of  calibration 
and  environment  are  of  a  comparable  magnitude  to  the  variations  among 
the  individual  models  within  a  type  category.  Additional  studies  are 
needed  to  quantify  the  following  effects  on  prediction  performance: 

•  The  level  of  the  initial  estimate. 

•  The  method  of  making  the  initial  estimate. 

e  The  method  of  making  subsequent  estimates. 

•  Alternatives  to  the  size  measure. 

One  analysis  that  is  suggested  by  the  present  study  is  the  determination 
of  the  influence  on  estimating  accuracy  of  the  use  of  size  of  code  as  an 
input.  Models  that  use  size  as  an  input  should  be  further  classified 
according  to  whether  the  code  is  decomposed  into  types  (e.g.  as  in  Boeing, 
Wolverton,  PRICE  S).  The  basis  for  comparison  in  each  case  should  be  the 
initial  estimate  because  in  most  models  a  process  different  from  the 
initial  one  is  used  to  manipulate  this  value  to  obtain  the  full  scope  of 
the  model  outputs.  It  is  necessary  to  evaluate  the  two  methods  indepen¬ 
dently. 
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The  DOD  Micro  Procedure,  Farr  and  Zagorski  and  Wolverton  models 
should  be  calibrated  to  the  evaluation  test  data.  This  would  provide 
seven  calibrated  models.  They  should  be  classified  as  follows: 

•  Size  of  code 

-  Types  of  code 

-  No  types  of  code 

•  No  size  of  code 

The  results  should  establish  to  a  higher  degree  of  confidence  than 
was  possible  in  the  present  study  the  difference  in  estimating  accuracy, 
if  any,  afforded  by  the  use  of  code  size  as  an  input.  They  would  also 
indicate  if  defining  the  type  of  code  to  be  written  increases  estimating 
accuracy.  Other  experiments  should  be  designed  to  test  the  performance  of 
different  methods  of  proceeding  to  other  cost  estimates. 

The  results  of  these  investigations  should  be  used  to  establish 
the  basic  attributes  of  model  structures  that  have  demonstrated  high 
accuracy..  When  this  has  been  accomplished  using  existing  models,  it  will 
be  possible  to  design  a  series  of  second  generation  models  based  on  the 
structures  that  have  performed  well  in  systematic  tests.  These  models 
will  be  coordinated  with  the  development  of  better  data  sets  which  will 
permit  more  complete  exploration  of  structures  and  more  comprehensive 
testing. 


7.2  DATA  DEFINITION  AND  COLLECTION 

Data  availability  and  quality  has  been  a  major  limiting  factor  in 
cost  model  development.  This  evaluation  has  indicated  the  importance  of 
data  definitions  to  the  interpretation  of  model  performance.  The 
recommended  direction  for  future  model  development  puts  additional  require 
ments  on  data.  The  Air  Force  has  recognized  the  need  for  software  data 
and  has  taken  a  major  step  in  the  establishment  of  the  data  repository  at 
the  Data  and  Analysis  Center  for  Software  (DACS).  We  recommend  that  data 
collection  efforts  continue  to  be  focussed  at  DACS  and  that  the  model 
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development  activities  be  used  as  the  basis  for  establishing  data  report¬ 
ing  requirements  under  software  development  contracts.  The  DACS  is  the 
ideal  catalyst  for  coordinating  the  dissemination  of  information  describ¬ 
ing  progress  toward  both  objectives. 

If  program  size  is  determined  to  be  an  important  factor  in  making 
accurate  estimates,  DACS  should  be  responsible  for  maintaining  standard 
definitions  under  which  project  data  could  be  reported  and  tabulated. 

The  same  should  be  done  for  any  other  inputs  or  outputs  required  for  cost 
estimating.  The  present  role  is  passive.  Given  a  productive  research 
program  the  definition  of  data  elements  should  become  active.  The  Air 
Force  should  take  charge  of  defining  its  needs  for  estimating  data. 

Software  data  reporting  should  become  an  integral  part  of  the 
contracting  process  much  as  operating  costs  are  now.  Items  and  formats 
should  be  defined  by  the  Air  Force  and  provided  routinely  by  the 
contractors.  Audits  should  be  possible  if  necessary  to  substantiate  the 
reported  values.  A  well-designed  data  reporting  scheme  (e.g.  [43]) 
should  not  be  burdensome  on  the  contractors  and  should  pay  for  itself  in 
better  planned  and  managed  system  development  projects. 

A  recommiended  data  collection  project  is  the  acquisition  of 
estimates  of  input  variables  during  the  life  cycle.  Tnt  results  in  the 
present  evaluation  were  obtained  using  actual  values  usually  recorded 
after  a  project  was  completed.  This  reflects  minimum  uncertainty  in  the 
inputs.  Therefore,  the  accuracy  of  the  different  models  is  higher  than 
it  would  be  if  we  included  the  precision  with  which  the  inputs  are  known. 
The  different  inputs  are  not  known  equally  well  at  the  different  times 
that  estimates  are,  made.  This  biases  the  results  in  favor  of  those 
models  which  use  inputs  that  are  known  accurately  only  late  in  the  develop¬ 
ment  cycle. 
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AEROSPACE  MODEL 


Description  of  the  Model 

The  model  was  developed  using  regression  techniques  applied  to  data 
from  software  development  projects  characterized  by  one-of-a-kind  computers, 
limited  support  software,  special  languages  and  severe  memory  size  and  speed 
requirements.  The  data  were  stratified  into  two  groups.  One  group  contained 
13  projects  for  the  development  of  real  time  software  identified  as  primarily 
large-scale  airborne  and  space  applications.  The  second  group  consisted  of 
7  operational  support  programs  presumably  without  the  size  and  speed  require¬ 
ments  of  the  first  group. 

The  model  description  is  not  clear  concerning  the  exact  composition  of 
the  estimate  of  effort  required  to  develop  the  software.  Only  the  total 
effort  is  estimated.  The  estimate  is  made  using  a  relationship  of  the  form; 

MM  =  a  (Instruction)*^ 

where  the  constants,  a  and  b,  are  determined  by  regression  analysis. 

The  estimating  relationships  are: 

Real  Time  Software 


MM  =  0.057  (1)°*^^ 

Support  Software 


whe»*e: 


MM  =  2,012 


(I) 


0.404 


MM  =  total  development  effort,  manmonths 

I  =  number  of  instructions  (independent 
of  language). 


Reference 

T.  G.  James,  Jr,,  "Software  Cost  Estimating  Methodology,"  Proceedings  ILEE 
VAECON  ’77,  May  77,  PP  26-27. 
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The  Boeing  Computer  Services  (BCS)  software  cost  model  estimates 
total  project  effort  from  a  table  of  productivity  rates  that  associates 
different  types  of  software.  The  rates  are  applied  to  the  sizes  of  the  de¬ 
livered  programs  to  obtain  estimates  of  the  direct  effort  required  to 
develop  the  programs. 

The  BCS  model  works  best  for  aerospace  types  of  systems.  The  most 
reliable  estimates  of  the  inputs  are  obtained  from  project  planners  who 
have  related  experience  and  a  good  knowledge  of  the  system  requirements. 

The  method  begins  with  decomposition  of  the  systems  into  functions  and 
modules.  This  requires  knowledge  and  a  certain  amount  of  component 
definition,  i.e.,  how  the  system  functions  will  be  performed  and  how  they 
should  best  be  divided  up  for  development. 

The  project  leader  is  asked  to  divide  the  system  according  to  its 
composition  among  the  following  types  of  software: 

•  Mathematical  Operations 

e  Report  Generation 

•  Logic  Operations 

•  Signal  Processing  or  Data  Reduction 

•  Real  Time  or  Executive  (also  Avionics  Interfacing) 

These  assignments  are  based  on  experienced  judgment  and  are  subjective. 

Then  an  estimate  is  made  of  the  total  number  of  del ivered  instructions. 
BCS  experience  indicates  that  come  care  must  be  taken  with  thi<^  estimate 
because  project  leaders  tend  to  think  in  terms  of  developed  code  and  not 
delivered  code.  But  they  have  found  it  necessary  to  make  this  distinction 
because  non-del ivered  code  such  as  test  drivers  is  normally  not  tested  and 
documented  as  thoroughly  as  delivered  code  and  consequently  requires 
substantially  different  development  resources. 


The  method  then  applies  different  productivity  rates  for  each  type 
of  software  to  obtain  development  effort.  The  rates  were  obtained 
empirically  from  over  two  dozen  sources.  Some  of  these,  for  example, 
include  Wolverton,  SDC,  TRW,  and  Boeing  projects  such  as  Lunar  Orbiter. 

Code  types  were  identified  and  productivity  was  obtained  from  total  project 
effort  data.  The  factors  were  obtained  by  curve  fitting.  The  results 
were  confirmed  by  knowledgeable  people.  The  productivity  rates  range 
from  6  to  40  manmonths  per  1000  source  statements. 

The  total  development  effort  is  divided  into  project  tasks  according 
to  a  fixed  schedule. 


%  OF  TOTAL 


Requirements  Definition 

5 

Design  and  Specification 

25 

Code  Preparation 

10 

Code  Checkout 

25 

Integration  and  Test 

25 

System  Test 

10 

The  resulting  values  are  then  adjusted  for  9  conditions: 

Reimplementation  of  existing  software 

Follow-on  contract 

Number  of  programmers 

Higher  order  language 

Macro  language 

On  line  code/data  entry 

On  line  debugging 

Poor  or  no  debug  tools 

Programming  experience 

The  adjustment  factors  for  each  condition  are  applied  to  the  appropriate 
task  efforts. 
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The  total  development  effort  includes  manmonths  for  all  direct 
personnel  (except  final  documentation)  and  first  line  supervision  (how¬ 
ever,  it  should  be  noted  that  many  of  the  sources  of  productivity  data  were 
non-Boeing  environments).  A  typical  group  would  have  a  supervisor  and  6 
to  10  programmers.  Larger  projects  with  higher  levels  of  management  would 
not  include  these  managers  in  the  total  effort  prediction. 

Outputs 

Raw  Development  Effort.  Man-months  to  fully  check  out,  test  and 
document  software  of  a  given  type  and  number  of  statements.  The  total 
development  effort  is  the  sum  of  the  development  efforts  required  for  each 
type  of  code.  This  is  a  raw  value  that  is  derived  from  estimates  (by  type 
of  software)  of  numbers  of  delivered  statements  of  new  code  divided  by 
productivity  rates.  The  rates  are  different  for  each  type  of  software.  The 
development  effort  is  allocated  among  six  development  phases.  The  man- 
months  in  each  phase  are  subsequently  adjusted  to  account  for  existing 
software,  higher,  order  language,  programming  experience  and  other  factors. 

Distributed  Raw  Development  Effort.  The  apportionment  of  the  Raw 
Development  man-months  according  to  fixed  percentages. 

Task  or  Development  Phase  Percent  of  Total  Effort 

Requirements  Definition  5 

Design  and  Specification  25 

Code  Preparation  10 

Code  Checkout  25 

Integration  and  Test  25 

System  Test  10 


Adjusted  Development  Effort.  The  man-months  of  development  effort  for 
each  software  type  separated  into  development  tasks  o'  phases  and  adjusted 
for  nine  product  and  environmental  factors. 


1.  Reimplementation  of  existing  software 

2.  Follow-on  contract  with  current  customer 

3.  Number  of  programmers 

4.  Higher  order  language 

5.  Macro  language 

6.  On-Line  code,  data  entry 

7.  On-Line  debugging 

8.  Poor  debugging  tools 

9.  Programming  experience. 

Computer  Time.  Two  lime  estimates  are  provided: 

•  Stand-alone  time  (dedicated  computer) 

t  Computer  resource  units  (not  defined) 

Stand-alone  or  dedicated  computer  availability  may  occur  in  develop¬ 
ments  utilizing  minicomputers  or  special  purpose  computers  that  are  totally 
devoted  to  u  given  application.  The, cost  of  the  computer  or  the  access  to 
computer  facilities  is  not  shared  with  any  other  software  development 
project. 

Computer  resource  units  are  used  to  measure  the  portion  of  a  large 
multi-user  computer  facility  appropriated  for  a  single  .ask  Usually,  the 
computer  operating  system  measures  the  amount  of  time  that  the  i ,sk  uses 
computer  memory,  different  peripheral  devices,  software  packages,  and  other 
resources  such  as  multipart  paper,  tapes,  etc.  and  uses  an  algorithm  to 
charge  the  user  according  to  the  portion  of  the  total  system  that  was  made 
available.  The  definition  of  a  computer  resource  unit  depends  on  the 
computer  equipment  configuration,  the  relative  costs  of  the  devices,  and 
the  method  ot  allocating  fixed  costs.  CRUs  are  not  in  general  comparable 
from  one  computer  installation  to  another. 
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Inputs 

Product  Related  Inputs 

Number  of  Statements.  The  number  of  statements  to  be  written  and 
delivered  to  the  customer.  The  count  includes  only  new  code  and  excludes, 
"test  drivers,  test  data  bases,  translators,  simulators,  etc.”  that  are 
written  but  not  delivered.  These  are  separated  from  the  delivered  code 
because  they  do  not  undergo  the  same  level  of  testing  and  documentation. 

They  are  accounted  for  in  the  estimating  procedure  by  a  subsequent 
adjustment  to  the  development  effort. 

The  number  of  statements  describes  executable  statements,  but  includes 
storage  defining  statements  (e.g.,  FORTRAN  COMMON).  An  allowance  should  be 
made  if  the  specifications  describe  unusually  severe  requirements  for 
commentary  within  the  code. 

The  number  of  statements  are  counted  according  to  •^ive  types  i-f 
software: 

•  Mathematical  Operations 

•  Report  Generation 

•  Logic  Operations 

•  Signal  Processor  or  Data  Reduction 

•  Real  Time,  Executive,  or  Avionics  Interfacing 

Resource  Related  Inputs.  The  following  outputs  are  used  to  modify  the 
estimates  of  the  development  effort  required  for  each  type  of  software 
They  are  used  to  identify  constant  adjustment  factors  that  are  associated 
with  the  different  development  phases.  Two  of  the  inputs.  Number  of 
Programmers  and  Programming  Experien'-e,  are  cardinal  numbers;  the  others 
are  either  applicable  to  the  planned  development  or  not. 

Number  of  Programmers.  Three  sizes  of  development  team  are  described: 

•  1-^ 

t  6-10 

•  More  than  20. 
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Each  team  size  is  associated  with  a  factor  that  increases  effort  with 
incr'asing  team  size. 

Program  ling  Experience.  The  relative  exoerience  of  the  group  in  the 
technical  descipline  being  programmed  is  identified  by  one  of  three  levels 

•  Entry-Level 

t  Moder..te 

•  High 

The  following  inputs  describe  factors  or  conditions  that  may  or  may 
not  be  appropriate  to  the  development. 

•  Reimplementation  of  existing  software 

•  Follow-On  contract  with  current  customer 

•  H’gher-Order  language  (seasoned  compiler) 

•  Macro-Language 

-  In  coding 

-  Forms  for  document 

•  On-Line  debugging 

e  Poor  (or  no)  debugging  tools  except  dumps 


Reference 

R.  N.  E.  Black,  R.  P.  Curnow,  R.  Katz,  M.D.  Gray,  BCS  Software  Product  on 
Data,  NTIS,  AD  A039852,  Mar  1977. 
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POD  MICRO  ESTIMATING  PROCEDURE 


Description  of  the  Model 

The  primary  estimating  relationship  comprising  the  DoD  Micro  Procedure 
can  be  described  as  the  ratio  of  a  factor  representing  the  software  to  be 
developed  or  changed  and  a  productivity  measure. 

The  model  form  suggests  that  effort  increases  directly  with  the  numbe*" 
of  input  and  output  configurations  operating  on  the  system  being  built. 

Effort  also  increases  with  the  number  of  routines  being  created  or  modified 
'  ’hted  by  their  difficulty.  The  total  effort  is  scaled  according  to  the 
amount  of  work  that  must  be  done  in  entirety  as  opposed  to  modification 
of  an  existing  system. 

The  number  of  days  needed  to  deliver  the  product  (effectively  the 
days  of  effort  per  unit  of  product)  depends  on  the  general  experience  and 
accomplishment  of  the  development  group  (measured  by  their  job  classifications) 
weighted  by  their  knowledge  of  the  problem  to  be  solved  relative  to  the 
knowledge  required.  One  other  factor  that  directly  affects  the  productivity 
is  the  ease  of  access  to  the  computer  (measured  by  turnaround  time). 

The  basic  form  of  the  estimating  relation  for  software  development 
time  is: 

Net  Development  Time  =  (Product)  -f  (Productivity) 

Where:  Product  is  a  measure  describing  the  effort  to  be  performed. 

Productivity  is  the  rate  of  creating  the  product  from 
the  application  of  personnel  time. 

Product  =  (Number  of  Formats  +  Weighted  Number  of  Functions) 

X  (Effort  Relative  to  a  New  Development) 
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The  terms  in  parentheses  along  with  the  following  terms  are  defined  in  the 
discussion  of  model  inputs  below: 

(Productivity)"^  *  (Work  Days  per  Unit  of  Product  for  a  Staff  with 
Average  Experience)  x  (Job  Knowledge  Required) 

X  (Job  Knowledge  Available) 

X  (Access) 

The  result  is  the  total  hours  required  for  code  development. 

Presumably  this  means  detailed  design,  coding,  and  unit  testing. 

Gross  Development  Time  =  (Net  Development  Time) 

X  (Other  System  Factor) 

X  (Non-Project  Factor  +  Lost  Time  Factor) 

A  value  of  1.8  is  recommended  for  the  other  system  factor.  This  factor 
represents  the  effort  needed  to  convert  the  code  development  time  to  total 
development  time.  This  value  is  representative  of  an  observed  range  from 
1.2  to  2.1.  Total  development  includes  analysis,  design,  coding,  testing 
and  documentation.  It  is  the  sum  of  the  project  direct  charges.  Whether 
this  includes  support  hours  for  clerical  and  other  functions  is  not  clear, 
but  any  given  organization  could  include  these  by  modifying  the  1.8  factor. 

The  net  development  time  accounts  for  the  time  lost  from  normal 
scheduled  working  hours  for  leave,  sickness,  holidays,  and  non-project 
assignments.  These  add  25  percent  to  the  total  development  time.  There 
is  also  a  10  percent  efficiency  factor  (coffee  breaks,  time  cards,  code 
rework,  etc.).  The  code  rework  should  probably  be  handled  elsewhere.  It 
is  probably  included  where  it  is  to  make  the  10  percent  palatable.  It  should 
be  Included  in  the  gross  size  adjustment  and  the  1.8  factor. 

The  effect  of  these  adjustments  is  to  estimate  the  number  of  personnel 
who  must  be  assigned  to  the  proj'ect  to  ensure  delivery  of  the  total 
development  hours.  These  factors  are  organizational  specific. 
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Although  the  resource  estimating  procedure  includes  weighting  factors 
for  the  input  and  output  formats  by  type  of  device  (see  subsequent  discussion), 
the  factors  have  a  value  of  one  in  each  case.  Therefore ,  the  model  describes 
a  linear  relationship  between  the  total  number  of  file  formats  and  the 
effort  required  to  implement  them.  It  may  be  that  future  versions  of  the 
model  will  weight  the  types  of  file  devices  differently.  Then  the  effort 
required  to  implement  a  report  format  may  be  different  from  the  effort 
required  for  a  card  format. 

Program  complexity,  which  is  the  second  term  in  the  product  measure, 
is  the  weighted  sum  of  the  functions  to  be  implemented.  The  weights 
depend  on  the  function  and  its  assumed  level  of  complexity.  The  weights 
range  from  1  for  a  simple  operating  system  control  language  change  to  12 
for  a  very  complex  edit-val idation  function. 

The  value  3  is  the  most  common  among  the  24  possible  function-complexity 
assignments.  If  the  function  types  are  equally  represented  in  programs, 
the  average  value  is  4. 

The  programmer/analyst  experience  factor  is  an  indication  of  the 
effect  of  experience  on  productivity.  Values  range  from  .75  to  2.75 
corresponding  to  a  lead  analyst  or  programmer  and  interns  respectively. 

Since  experience  is  not  evenly  distributed  over  a  group  of  programmers  and 
analysts,  the  following  groups  was  hypothesized  in  order  to  obtain  an 
average  or  representative  value  for  the  experience  factor. 


Experience 

Number 
in  Group 

Factor 

Weighted 

Sum 

Lead 

1 

.75 

.75 

Senior 

2 

1 .25 

2.50 

Journeyman 

4 

1  .75 

7.00 

Nominal 

8 

2.25 

18.00 

Intern 

_5 

2.75 

13.75 

20 

42.00 

Average  Value  =  42  +  20  «  2.1 


No  definitions  are  provided  for  the  10  job  classifications. 

The  job  knowledge  and  turn-arojnd  time  factors  are  self-explanatory. 

The  System  Factof  adjusts  the  product  development  effort  to  account 
for  work  already  done.  The  product  measure  resulting  from  the  format 
count  and  the  program  complexity  value  is  the  same  whether  the  system  is 
being  developed  in  its  entirety  or  it  is  a  modification  to  an  existing 
system.  The  system  factor  has  the  effect  of  modifying  the  product  value 
to  account  for  less  than  total  development. 

Seven  levels  of  change  are  described  by  the  System  Factor.  The  values 
range  from  2  for  a  new  development  to  8  for  an  operating  systems  control 
language  change. 

For  a  new  system  development  the  2  in  the  primary  estimating  equation 
is  divided  by  a  System  Factor  value  of  2  and  the  product  measure  is  un¬ 
changed.  Consequently,  the  System  Factor  values  describing  lesser  amounts 
of  new  development  have  larger  values  and  are  portions  of  2.  The  effect 
of  the  System  Factor  on  the  product  measure  is  summarized  as  follows: 


Type  of  Effort 

System  Factor 

Effort  Relative  t( 
a  New  Development 

New  Development 

2 

1 .00 

Major  Change 

3 

.67 

Major  Modification 

4 

.50 

Minor  Modification 

5 

.40 

Maintenance 

6 

.33 

Minor  Technical  Change 

7 

.29 

Operating  Systems 

Control  Language  Change 

8 

.25 
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In  order  to  get  a  feel  for  the  relative  magnitudes  of  the  components 
of  the  Micro  Estimating  Procedure,  consider  the  following  example. 

Number  of  I/O  formats  =  10 
Number  of  functions  =  20 
Average  compli-xity  factor  =  4. 

New  Development 

Product  =  (Number  of  Formats  -t-  Weighted  Number  of  Functions) 

X  (Effort  Related  to  a  New  Development) 

Product  =  (10  +  4  X  20)  X  2^2  =  90 

Experience  =  2.  (See  above  for  computation) 

Job  knowledge  required  =  1.0 
Job  knowledge  available  =1.0 
Access  =  =1.0 

(Product! vi ty)"^  =  (Work  Days  per  Unit  of  Product  for  a  Staff  with 
Average  Experience) 

X  (Job  Knowledge  Required) 

X  (Job  Knowledge  Available) 

X  (Access) 

=  2.0  X  1 .0  X  1 .0  X  1  .0  =  2.0 

Net  Development  Time  (Product)  x  (Productivity)'^ 

=  90  X  2.0  =  180  Man-Days 

If  the  effort  was  a  major  modification  (System  Factor  =  4),  the 
Product  value  becomes: 

Product  =  (10  +  4  X  20)  x  2-5-4  =  45 
and 

Net  Development  Time  =  45  x  2.0  =  90  Man-Days 
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If  the  Job  Knowledge  Required  1s  "Detailed"  (Factor  »  1,5)  and  the 
Job  Knowledge  Available  Is  "Limited"  (Factor  *  1.5),  the  productivity 
becomes: 

(Producti vi ty)"^  =  2.0  x  1 .5  x  1 .5  x  1 .0  *  4.5 
then  for  the  major  modification: 

Net  Development  Effort  =  45  x  4,5  *  202,5  Man-Days 

Outputs 

The  primary  output  (i.e.,  the  output  that  is  sensitive  or  controlled 
by  project  variables  as  opposed  to  the  subsequent  step  which  is  a  fixed 
allocation)  is:  Gross  Development  Time  (man-days).  Gross  Development  Time 
includes: 

•  Nonproject  time  (individual  assigned  to  project  but  busy  with  non¬ 
project  tasks,  e.g.,  training,  non-product  administrative  duties, 
etc,,  and  vacation  and  holidays) 

•  Wasted  or  lost  time 

Therefore,  Gross  Oevelpment  Time  describes  the  staffing  level  that  will 
result  in  a  needed  amount  of  development  time.  The  latter  is  predicted  by 
program  and  project  characteristics. 

The  secondary  outputs  (i.e.,  those  derived  by  applying  fixed  values 
to  the  primary  output  are: 

•  Effort  by  project  phase 

•  Total  development  cost 

The  project  phases  are: 

•  Review  and  analysis 

•  Design 

•  Programming 

•  Testing 

•  Documentation 
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Gross  Development  Time  includes: 

Analysis  of  present  methods 

Design  of  the  new/changed  system 

Develop  the  system's  support 

Program  design 

Program  development 

Program  testing 

System  testing 

Installation  and  conversion 

Staff  training 

Project  officer 

System  manager 

Technical  managers 

Support  personnel 

Documentation 


Inputs 

Product  Related  Inputs.  The  software  is  described  by  the  numbers  of 
types  of  items  it  processes  and  the  numbers  of  functions  it  includes.  The 
functions  are  described  according  to  type  and  complexity.  The  result 
is  two  product  descriptors:  one  measures  the  size  of  the  input/output 
processing  to  be  executed  by  the  system;  the  other  is  a  measure  of  the 
number  and  difficulty  of  the  functions  to  be  performed. 

Input  File  Formats.  The  number  of  different  -^ormats  to  be  read  by 
the  system  are  counted  and  added  together.  The  model  asks  for  numbers 
of  card,  tape,  disk,  and  screen  formats  separately,  but  since  the  weighting 
factor  is  always  one,  there  is  no  distinction  made  among  them  regarding 
the  effort  involved  to  implement  them. 

Output  File  Formats.  The  formats  output  by  the  system  are  totaled. 

The  same  entries  as  for  the  inputs  are  requested  plus  the  number  of  report 
formats.  As  in  the  case  of  the  inputs,  the  weighting  factor  for  the 
different  types  of  output  is  always  one,  so  there  is  no  reason  to  differentiate. 
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Program  Complexity.  The  total  program  complexity  measure  is  computed 
by  a  weighted  sum  of  the  number  of  processing  functions  of  given  types.  Each 
function  is  characterized  as  simple,  complex,  or  very  complex.  The  processing 
functions  are: 

e  Edit  Validation 

•  Table  Look-Up  (Internal  or  External) 

•  Calculations 

•  Sort/Merge  Process 

•  Internal  Data  Manipulation 

•  File  Search 

•  Utilities  or  Subroutines 

•  Operating  Systems  Control  Language 

Job  Knowledge  Required.  The  amount  of  knowledge  required  to  implement 
or  change  a  system  has  a  direct  effect  on  the  number  of  hours  required  to 
accomplish  the  project.  A  system  that  requires  very  detailed  knowledge  will 
require  more  effort  than  one  that  can  be  accomplished  with  limited  knowledge. 
This  parameter  is  paired  with  the  job  knowledge  available  factor  described 
below  to  describe  the  relative  influence  on  productivity.  Three  job 
knowledge  levels  are  used:  Limited,  General,  Detailed. 

System  Factor.  The  effort  required  to  complete  a  system  development 
or  change  project  of  given  complexity  depends  on  the  state  of  the  system. 

That  is,  the  work  required  to  change  three  file  formats  is  less  than  the 
work  required  to  develop  a  system  with  three  file  formats,  all  other 
factors  being  equal.  The  System  Factor  describes  the  level  of  effort 
being  undertaken.  Seven  levels  are  described: 

t  System  development 

•  Major  changes 

•  Major  modification 

•  Minor  modi fication 

•  Maintenance 

t  Minor  technical  change 

•  Operating  systems  control  language 
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Resource  Related  Inputs 

Proqrammer/ftnalyst  Experience  Available.  The  available  experience 
measure  is  an  effective  productivity  indicator.  It  quantifies  the  rate 
at  which  the  product  can  be  produced  in  terms  of  the  job  classification 
of  the  staff  available  for  assignment  to  the  system  development.  Two  data 
processing  personnel  classifications;  Analyst  and  Programmer,  are  tabulated 
according  to  five  levels  of  experience;  Lead,  Senior,  Journeyman,  Nominal, 
and  Intern.  Weights  are  associated  with  the  difference  experience  levels. 

The  result  is  a  weighted  average  productivity  factor. 

Job  Knowledge  Available.  This  factor  has  the  effect  of  describing 
the  change  in  productivity  associated  with  the  level  of  knowledge  about  the 
rtork  to  be  performed  that  exists  among  the  persons  available  for  assignment. 

;t  works  together  with  the  Job  Knowledge  Required  factor  described  above 
to  quantify  the  effect  of  the  knowledge  of  the  system  required  compared 
to  that  available  on  the  time  required  to  complete  the  work.  In  general, 
the  effect  of  the  combined  factors  is  to  increase  the  development  manhours 
if  the  need  exceeds  the  available  and  decrease  the  hours  if  the  available 
exceeds  the  need.  Three  levels  of  job  knowledge  availability  are  specified; 
Limited,  General,  and  Detailed. 

Program  Turn-Around  Time.  The  effect  of  computer  access  on  productivity 
is  described  by  four  levels  of  average  turn-around  time; 

•  Interactive  terminal 

•  More  than  one  run  per  day 

•  One  run  per  day 

•  Less  than  one  run  per  day. 


Re  r'e*  ence 

■t'ndard  Dou  ADP  Resource  Estimating  Procedure  (REP)  for  Software  Development 
.Draft),  Dept,  of  Defense,  Sept  1979. 
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Doty  Associate*.. ,  Inc. 

Description  of  the  Model 

The  model  is  actually  a  set  of  15  estimating  relationships.  Eacn 
one  to  be  used  for  a  given  type  of  software  and  software  life  cycle  phase. 
Equations  have  been  derived  empirically  using  regression  analysis  for 
the  following  types  of  software: 

•  Command  and  Control 

•  Scientific 

•  Business 

•  Utility 

The  development  effort  for  softw,'5'^e  representing  each  of  the  appli¬ 
cation  types  may  be  estimated  using  one  of  three  different  relaticr.si’.’ps.  An 
additional  three  are  given  that  are  applicable  to  all  types  of  software.  These 
equations  are  to  be  used  "when  the  application  cannot  be  categorized  or 
is  different  than  the  categories  noted".  The  procedure  specifies  that 
when  a  software  system  is  made  up  of  subsystems  that  are  of  different 
types,  the  total  size  should  be  divided  into  the  four  categories  and 
the  appropriate  estimating  equation  used  for  each  one.  Then  the  individual 
manmonths  are  summed  to  give  a  total  system  development  effort.  The  three 
equations  are  divided  into  size  measure  (lines  of  source  code  or  words  or 
object  instructions)  and  the  life  cycle  phase  in  which  the  estimate  is 
made  (Concept  Formulation  and  all  others).  If  the  estimate  is  to  be  made 
using  the  words  of  object  instructions,  the  same  equation  is  used  in  all 
life  cycle  phases.  Similarly,  for  estimating  large  systems  (more  than 
10,000  lines)  using  lines  of  source  code  requires  the  use  of  a  different 
equation  in  the  Concept  Formulation  Phase  than  in  the  other  life  cycle 
phases. 


The  use  of  the  different  equations  can  be  described  as  follows 
(A,  B,  and  C  refer  to  the  three  different  relationships). 
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i  SOFTWARE 

DESCRIPTION 

LIFE  CYCLE  PHASE  j 

CONCEPT  OTHERS  ! 

1 

- - - * 

WORDS  OF  OBJECT  CODE 

A 

- 1 

A  j 

LINES  OF  SOURCc  CODE 

( 

LARGE  SYSTEM  >  lOK  LINES  | 

. 

i 

1 

1 

B  1 

i 

SMALL  SYSTEM  <  lOK  LINES  j 

! 

—  —  ■  —  i 

C  i 

1 

The  forms  of  the  estimating  relationships  are  similar.  Equat'or 
''  and  B  are  of  the  form: 

MM  =  a 

where  MM  =  Manmonths  of  development  effort. 

I  =  either  wcrds  of  object  code  (A)  or  lines  of 
executable  source  code  (B). 

a,b  =  Constants  obtained  empirically. 

Equation  C  has  the  form: 

MM  =  c  I  T  f 
j  =  l  j 

Where  'f.!  =  3  set  of  parameters  describing  the  development 
^  environment. 

c,d  =  constants  obtained  empirically 
Values  of  the  constants  to  be  used  for  different  types  of  applications 
are  given  in  Tables  A-1  and  A-2. 

The  followina  guidelines  are  presented  for  selecting  the  proper 
estimating  relationship. 
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•  In  Concept  Formulation,  if  the  size  of  the  program  in  object 
code  is  known,  use  the  object  code  estimators.  They  will  give 
more  accurate  estimates  of  manpower  requirements.* 

•  If  accurate  estimates  of  manpower  requirements  are  required 
in  the  Analysis  and  Design  and  subsequent  phases  of  develop¬ 
ment,  use  equation  B,  in  source  code ,  for  programs  of  I  ^  10,000 
and  equation  in  source  code,  for  programs  with  I  <  10,000. 

•  For  budgetary  purposes,  use  the  equation  that  gives  the  higher 
estimate. 

Development  time  is  estimated  using  the  equation 

.  _  10001 _ 

92.25  +  2331  ’  ° 

where  0  =  Reasonable  development  time  in  months 
I  =  Number  of  delivered  object  instructions. 


This  relationship  was  obtained  using  regression  on  data  describing 
74  development  projects.  The  time  estimate  should  describe  "customary" 
distributing  of  effort  over  time  that  is,  it  should  avoid  extremes  of 
project  time  compression  or  expansion. 


according  to  one  of  the  authors,  size  of  the  object  code  is  recommended 
over  size  of  the  source  code  as  an  estimator  because  most  of  the  deve¬ 
lopments  of  interest  to  the  study  sponsors  are  in  the  area  of  command 
and  control  and  scientific  systems.  In  these  areas  both  estimators  have 
similar  precision,  but  the  authors  believe  that  object  code  is  more  commonly 
used  and  understood  and  is,  therefore,  a  more  reliable  estimator. 
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It  should  be  noted  that  a  large  portion  of  the  documentation 
accompanying  the  description  of  the  DAI  estimating  procedures  is  devoted 
to  discussions  of  factors  that  are  believed  to  influence  the  cost  of  soft¬ 
ware  development.  These  factors  are  classified  according  to  aspects  of 
software  and  its  development  environment.  The  factors  are  grouped  according 
to  the  following  "domains": 

•  Requirements 

•  System  Architecture/Engineering 

•  Management. 

Outputs 

Cost  of  Software  Development 

The  estimate  of  total  development  cost  is  based  on  several  relation¬ 
ships  that  portion  the  cost  into  components  that  can  be  estimated  by 
applying  available  ratios  to  other  costs  and  factors  such  as  overhead 
and  administrative  costs.  By  the  proper  use  of  relevant  values  for  these 
factors  the  relationships  can  represent  either  government  in-house  costs 
or  contractor  development  costs.  A  method  is  described  for  time  phasing 
the  expenditure  that  is  said  to  satisfy  the  requirements  of  DoD  Directive 
5000.1 . 


The  procedure  identifies  costs  that  are  incurred  by  the  governmer* 
during  all  phases  of  the  software  life  cycle  except  Operation  and  Support. 
The  total  development  cost  includes: 

r  »  r  +  c  +  C 
^  ‘‘CF  ‘'VAL  ^FSD 

where  C  ■  Development  Cost 

Cj,p  ■  Conceptual  Phase  Cost 
C^al  *  Validation  Phase  Cost 
Cpjjj  “  Full  Scale  Development  Cost, 
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Information  is  inclnded  that  relates  the  government  cost  to  the 
contractor's  full  scale  development  cost.  This  cost  is  the  one  developed 
by  the  formal  software  cost  estimating  procedure. 


The  cost  of  development  is  divided  into  primary  and  secondary 
costs,  thus: 


Cp  + 


where 


Then, 


Cp  “  Cost  of  Development 
Cp  »  Primary  Cost  (Manpower) 

Cj  “  Secondary  Cost  (Computer, 
Documentation,  Etc.) 


where 

and 


Cp  *  MM(Cg) 

MM  »  Total  Development  Man-Months 
Cg  =*  Average  Labor  Cost 


Therefore: 


where 


n 

Cp  "  ^  C-  -  kCp 
i=l 

Cp  =  (MM)  Cg  (i  +  k) 

k  ■  Ratio  of  Secondary  to  Primary  Costs 
(•.075) 


The  total  software  development  cost  (does  not  Include  government 
Conceptual  and  Validation  Phase  costs)  includes  the  costs  of: 


Analysis 

Design 
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•  Code 

t  Debug 

•  Test  and  Checkout 

and  is  proportional  to  the  total  rnan-months  of  development  effort. 

Total  Development  Man-Months 

This  is  the  primary  output  variable.  It  Is  the  basis  for  the  total 
development  cost  estimate  and  it  is  the  value  from  which  the  distribution 
of  effort  by  life  cycle  phase  is  derived.  The  hours  include  those 
directly  related  to  the  development  of  the  software  system.  They  include 
the  direct  hours  needed  for: 

Analysis  -  interpreting  the  system  requirements  and  producing 
viable  alternative  system  concepts 
Design  -  preparing  detailed  designs  of  the  data  processing 
system  and  the  individual  programs 
Coding  and  Debugging  -  writing  individual  modules  and  programs 
and  performing  individual  tests 
Testing  and  Checkout  -  integrating  the  individual  subsystems 
into  a  complete  system  and  conducting  prescribed  tests 
on  the  entire  system. 

The  discussion  of  the  model  does  not  indicate  the  extent  that 
iuoport  and  management  hours  are  included  in  the  total.  Also,  there  may 
ee  some  question  about  the  activities  associated  with  concept  development 
',e.g.,  is  the  test  plan  furnished  by  the  government  following  the  validation 
onase  or  is  it  developed  as  part  of  the  project).  As  in  many  cost  estimating 
situations,  the  line  between  concept  analysis  and  the  evaluation  of  solutions 
lo  selected  concepts  is  hazy. 

Although  the  DAI  documentation  and  discussions  with  the  authors 
indicate  that  the  model  includes  integrated  system  testing,  it  appears 
that  this  effort  is  not  included  in  the  original  SDC  data  which  was  the 
sasis  for  the  curve  fits.  (76%  of  the  SDC  data  points  describe  programs 
that  do  not  interface  with  any  other  programs). 
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Software  Development  Time 

A  nominal  development  time  is  presented  that  implies  "customary 
manloading".  That  is,  the  schedule  does  not  reflect  either  crash 
projects  or  allow  for  unnecessary  delays. 

Distribution  of  Development  Effort 

The  expenditure  of  time  and  effort  associated  with  major  project 
milestones  is  given  for  small  projects  (one  level  of  supervision)  and 
large  projects  (more  than  one  level  of  supervision).  The  distributions 
are  for  nominal  projects  and  do  not  allow  for  any  possible  acceleration 
or  delay  of  the  completion  o-^  project. 


Development  Milestones 

Desirable  Distribution  of  Effort 

First  Level  Project 

r— - 

Second  Level  Project 

Schedule 

Expenditiire 

Schedule 

Expenditure 

Complete  System  Design  (PDR) 

WBM 

5% 

mM 

Complete  Package  Design 

27% 

Complete  Unit  Design  (CDR) 

36% 

mssm 

Complete  Unit  Code 

54% 

49% 

50% 

Complete  Unit  Debug 

64% 

59% 

57% 

Complete  Package  Test 

81% 

78% 

80% 

73% 

Complete  System  Test 

100% 

100% 

100% 

100% 

Inputs 

Program  Size 

DAI  has  been  very  careful  to  describe  the  size  variables  which  are 
the  primary  Inputs  to  the  estimating  equations.  This  should  help  make 
more  reliable  estimates  using  the  relationships.  However,  we  should 
point  out  that  the  respondents  to  the  original  SDC  questionnaire  were  not 
so  well  directed  and  it  may  be  necessary  when  analyzing  the  structure  of 
the  model  as  It  relates  to  prediction  accuracy  that  significant  errors 
may  have  been  introduced  by  this  failure  to  be  specific.  The  DAI  model 
may  not  overcome  what  are  ipre»-ert  ’•'mitations  '’n  the  data. 
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The  DAI  procedure  calls  for  several  estimates  in  support  of  the 
OSARC  process.  It  recognizes  that  the  best  estimates  of  program  size  are 
obtained  later  in  the  development  cycle.  It  suggests,  then,  that  the 
interpretation  of  the  program  size  changes  during  the  life  cycle  and  that 
associated  with  the  changes  are  increases  in  estimating  accuracy.  The 
report  describes  how  the  knowledge  of  the  size  estimator  changes  during 
the  life  cycle  and  how  this  affects  the  estimating  precision.  The 
precision  associated  with  the  different  size  measures  during  the  system 
development  li"‘e  cycle  is  as  follows. 


1. 


2. 


3. 


4. 


Software  estimate 

When 

Sizing  basis 

%  Error 

Initial  program 
budgetary  estimate 

Conceptual  phase 

Total  object  code 

up  to  200%* 

Independent  program 
validation  cost 
estimate 

Validation  prior 
to  RFP  release 

Total  object  minus 
data  areas 
(Executable  Code) 

up  to  100% 

Independent  FSO 
cost  estimate 

Completion  of 
system  Spec 
through  PDR 

Total  object  minus 
data  areas  with 
adjustments  for 
reusable  code 

up  to  75% 

Update  of  FSD 
cost  estimate 

PDR  through 
remainder  of 
development 

Total  source  code 

up  to  50% , 
in¥>roving 
to  zero  at 

.  completion 

*The  actual  may  be  200  percent  of  the  estimated  or  the  estimated  may  be  200 
percent  of  the  actual. 
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Code  that  Is  developed  as  part  of  the  project  but  is  not  delivered 
to  the  customer  is  a  source  of  variation  in  the  estimate  of  the  system 
size  and  must  be  considered.  However,  no  guidance  is  provided  for  making 
any  adjustment  other  than  citing  that  the  SDC  data  showed  delivered  code 
to  average  77  percent  of  the  developed  code  with  a  standard  error  of 
30  percent. 

Allowance  must  also  be  made  for  support  software  development 
especially  when  working  with  new  hardware. 

Total  Object  Words 

During  the  Conceptual  Phase  when  very  little  is  known  about  the 
system  to  be  developed,  the  initial  estimate  is  made  using  the  analyst's 
judgement  (usually  by  analogy  with  previously  developed  systems,  but 
other  methods  are  possible)  of  the  number  of  object  words  occupied  by 
"every  program  needed  to  run  and  maintain  the  system  in  the  field".  This 
measure  is  obtainable  from  listings  of  computer  system  routines  that 
build  executable  programs  from  the  ootput  of  the  compiler.  Taking 
values  from  systems  similar  to  the  one  being  planned  can  provide  a  basis 
for  estimating  the  value.  Care  should  be  taken,  however,  when  program 
overlays  are  involved.  Also,  extensive  use  of  standard  library  routines 
can  greatly  increase  the  words  of  object  program  size  and  not  be  repre¬ 
sentative  of  a  comparable  increase  in  development  effort. 

Total  Object  Words  Minus  Data  Areas 

The  memory  space  occupied  by  an  executable  program  is  composed  of  loca¬ 
tions  containing  instructions  and  locations  reserved  for  the  data  upon  which 
the  program  will  operate.  Sometimes  the  data  storage  areas  are  signifi¬ 
cantly  larger  than  the  area  occupied  by  the  actual  instructions.  CAI 
suggests  that  the  effort  required  to  develop  the  programs  is  more  closely 
related  to  the  size  of  the  instruction  space  than  to  the  size  of  the 
combined  data  and  instruction  storage.  However,  as  in  the  case  of  the 
total  object  words,  there  is  no  evidence  of  this  distinction  being  made 
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in  the  original  derivation  of  the  estimating  procedures.  Also,  there  is 
no  guidance  provided  on  how  to  apply  the  additional  information  when 
preparing  cost  estimates.  Some  computer  system  executive  processing 
routines  provide  this  information.  However,  many  don't  and,  therefore, 
it  would  be  very  difficult  to  obtain  comparable  historical  information 
to  guide  new  estimates. 

New  Object  Words  Minus  Data  Areas 

Only  the  writing  of  new  code  contributes  to  the  software  develop¬ 
ment  effort  (if  code  written  to  modify  existing  modules  is  counted  as 
new  code).  To  account  for  the  work  done  to  adapt  existing  code  to  a  new 
system,  which  includes  analyzing  the  code  and  deciding  how  to  modify  it, 
any  existing  module  that  will  result  is  less  than  50  percent  utilization 
of  existing  code  is  considered  to  be  entirely  new. 

New  Source  Lines 

Counts  of  new  source  lines  written  (whether  in  a  higher  order  or 
machine  oriented  language)  can  be  obtained  from  compiler  listings,  measuring 
card  decks  or  text  editors.  It  is  one  of  the  easiest  measures  of  size 
to  obtain.  As  in  the  previous  case,  modules  containing  less  than  50  per¬ 
cent  reused  code  are  considered  to  be  new. 

Development  Environment 

For  estimates  made  using  lines  of  source  code  where  the  size  is  less 
than  10,000  lines,  the  estimating  relationship  includes  a  number  of  fac¬ 
tors  describing  the  development  environment.  These  are  included  in  the 
estimate  when  the  indicated  item  is  to  be  part  of  the  development  process. 
Detailed  definitions  of  the  factors  are  presented  in  an  Appendix. 

f.|  Special  Display 

f2  Detailed  Definition  of  Operational  Requirements 

Change  to  Operational  Requirements 

f^  Real  Time  Operation 
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fg  CPU  Memory  Constraint 

fg  CPU  Time  Constraint 

fy  First  SW  Developed  on  CPU 

fg  Concurrent  Development  of  ADP  Hardware 

fg  Time  Share  Versus  Batch  Processing  in  Development 

f^Q  Developer  Using  Computer  at  Another  Facility 

f^^  Development  at  Operational  Site 

fl2  Development  Computer  Different  from  Target  Computer 

f^g  Development  at  More  than  One  Site 

fl4  Programmer  Access  to  Computer 

After  analyzing  the  method  used  by  DAI  to  obtain  their  estimating 
relationships  and  after  comparing  their  definitions  of  input  and  output 
variables  with  the  original  sources  of  data,  it  is  clear  that  there  are 
discrepancies  between  the  way  the  data  are  being  applied  and  what  they 
originally  represented.  DAI  does  not  explicitly  justify  their  approach 
but  their  presentation  of  the  estimating  procedure  does  give  consideration 
to  errors  arising  from  differing  definitions  of  the  variables. 

DAI  seems  to  be  saying  that  consistent  use  of  the  estimating 
procedures  regardless  of  how  they  were  obtained  will  produce  results  with 
at  least  a  predictable  error.  That  is,  knowing  the  range  of  error  that 
can  occur  because  of  differences  in  definitions  and  ability  to  predict 
the  input  variables  will,  when  applied  to  the  given  estimating  relation¬ 
ships,  produce  estimates  with  precision  that  is  in  accordance  with 
previous  experience.  DAI  further  substantiates  the  approach  of  throwing 
all  the  error  into  the  ability  to  define  the  input  by  presenting  standard 
error  values  for  the  size  variables  at  different  times  in  the  life  cycle. 

References 
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FARR  AND  ZAGORSKI  MODEL 
Description  of  the  Model 


System  Development  Corporation  completed  several  projects  for  the 
Air  Force,  Electronic  Systems  Division  in  which  they  attempted  to  develop 
methods  for  predicting  the  cost  of  software  developnent.  The  Farr  and 
Zagorski  model  represent  an  intermediate  stage  in  the  program. 

Using  historical  data  from  internal  projects  and  from  other  organizations, 
the  SDC  team  systematically  tested  over  100  variables  to  learn  if  they  were  sat¬ 
isfactory  predictors  of  program  design,  coding  and  debugging  effort. 

Farr  and  Zagorski  published  three  equations*  which  were  determined 
to  be  the  best  predictors  tested  up  to  that  time. 

MM  =  2.7X^  +  121X2 

MM  =  2.8Xg  +  1.3X7  +  33X3  -  17Xg  +  lOXg  +  X^q  -  188  (2) 

MM  =  8.4X^^  +  1.8X^2  ^  •’’h  *  (3) 

Definition  of  Output 

MM  is  the  number  of  manmonths  needed  to  design,  code  and  debug  a 
single  program.  The  effort  begins  when  a  programmer  or  analyst  is  given  a 
complete  operational  specification  for  a  program  and  it  ends  when  the  program 
is  released  for  integrated  system  testing. 

Definitions  of  Inputs 

X.|  =  number  of  instructions  in  original  estimate  (in  thousands) 

X2  “  subjective  rating  of  information  system  complexity  (scale  1-') 

X3  =  number  of  document  types  delivered  to  customer 

X^  =  nunber  of  document  types  for  internal  use 


*  L.  Farr  and  H.  J.  Zagorski,  "Quantitative  Analysis  of  Computer  Programmi nq 
Cost  Factors:  A  Progress  Repor',  '  Proreedings  ICC  Symposium,  Rome,  1965 


=  number  of  computer  words  needed  to  store  program  data  (log-iQ) 

Xg  =  number  of  instructions  in  delivered  program  (in  thousands) 

tj  =  number  of  man-miles  for  travel  (in  thousands) 

Xg  =  system  programmer  experience  (average  of  total  years  of  experience 
with  the  computer,  language,  and  application) 

Xg  =  number  of  display  consoles 

X-jQ  =  percent  of  instructions  new  to  this  program  (not  re-used  from 
previous  versions) 

X^^  =  number  of  instructions  to  perform  decision  functions  (in  thousands) 

X^2  =  number  of  instructions  to  perform  nondecision  functions 
(in  thousands) 

X-jg  =  programmer  experience  with  this  application  (average  number  of 
years) . 

References 

L.  Farr,  H.  J.  Zagorski ,  Factors  that  Affect  the  Cost  of  Computer  Prcgramming, 
Vol  II,  A  Quantitative  Analysis,  NTIS,  ADeOZSAe,  Sep  1964.  Quantitative 
Software  Models,  Data  and  Analysis  tenter  for  Software,  SRR-1,  Mar  1979 
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PRICE  S 

Description  of  the  Model 

PRICE  S  is  a  proprietary  software  cost  estimating  model  developed 
and  maintained  by  PRICE  Systems  Division  of  RCA,  Cherry  Hill,  New  Jersey. 

It  is  installed  on  the  On-Line  Systems,  Inc.  time  sharing  network  and  may 
be  accessed  using  several  different  types  of  terminal  devices.  On-Line 
Systems  provides  local  dial-up  service  in  many  cities  throughout  the 
United  States. 

PRICE  S  estimates  software  development  costs  by  systematically 
adjusting  the  estimate  of  an  initial  element  of  the  cost  structure.  The 
initial  estimate  is  a  function  of  the  size  of  the  system  to  be  developed 
and  several  other  parameters  describing  the  characteristics  of  the  software 
and  the  development  environment. 

According  to  Frank  Freiman,  the  creator  of  PRICE  S,  the  model  design 
reflects  an  understanding  of  why  costs  attain  their  values.  This  is  in 
contrast  with  estimating  costs  by  fitting  hypothesized  relationships  to  histori¬ 
cal  data.  The  PRICE  S  developers  contend  that  software  development  proj¬ 
ects  are  complex  undertakings  whose  costs  are  influenced  by  a  inultipl  icity 
of  factors  too  numerous  to  analyze  and  often  impossible  to  measure. 

Consequently,  no  two  projects  are  alike.  This  makes  it  impossible  to 
identify  common  characteristics  among  past  projects  to  be  used  for  esti¬ 
mating  new  ones.  It  also  means  that  no  past  project  is  exactly  relevant 
to  any  future  one. 

Freiman  contends  that  a  manager's  perception  of  what  a  project  should 
cost  actually  determines  its  estimated  cost.  Therefore,  the  proper  formulation 
of  a  cost  estimating  model  is  in  terms  of  perceived  relationships  between 
coit  and  aspects  of  software  development  that  knowledgable  managers  believe 
determine  cost.  PRICE  S  estimates  the  cost  of  developing  code  using  hypo- 
tnesized  relationships  that  were  subsequently  supported  by  the  opinions 
individual  managers.  The  primary  relationships  describe: 
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•  The  cost  required  to  produce  programs. 

•  The  effect  on  cost  of  changing  development  time. 

•  The  comparative  costs  of  the  development  cycle  elements. 

In  addition  to  the  primary  relationships  there  are  many  secondary 
ones  which  may  be  subjective  or  empirical.  The  ultimate  justification  for 
all  the  hypotheses  is  the  model's  estimating  performance. 

The  relationship  between  the  cost  of  producing  a  program  and  attri¬ 
butes  of  the  code  was  originally  conceived  using  an  analogy.  The  analogy 
is  between  the  force  required  to  move  an  object  and  the  effort  needed  to 
write  software.  The  mass  of  an  object  is  the  product  of  its  volume  and 
density.  The  property  of  mass  is  a  function  of  its  spatial  concentration 
and  its  extent  or  volume.  In  a  similar  manner  the  cost  or  producing  code 
is  related  to  the  product  of  its  density  or  difficulty  and  its  extent  or 
size.  Furthermore,  the  cost  relationship  should  satisfy  the  empirical 
observation  that  the  cost  per  pound  of  a  wide  range  of  items  decreases  as 
the  weight  increases. 

In  PRICE  S  the  difficulty  associated  with  a  computer  program  is 
represented  by  a  parameter  called  its  APPLICATION  (APPL).  The  value  of 
APPL  is  small  for  easy  to  write  codes  such  as  mathematical  applications 
(APPL =0.86)  and  is  large  for  interactive  operations  and  operating  systems 
(APPL=10.95). 

The  size  of  the  system  is  measured  in  terms  of  the  number  of  machine 
level  instructions  represented  by  the  developed  code.  PRICE  S  uses  the 
mnemonic  INST  to  represent  the  size  of  the  software  system. 

The  product  (APPL)  X  (INST)  is  termed  the  weight  or  mass  of  the 
system  and  represents  a  portion  of  the  development  effort. 
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For  a  given  system  the  development  effort  Is  affected  by  the  develop¬ 
ment  time  as  well  as  the  type  of  application.  A  system  with  an  accelerated 
schedule  becomes  more  difficult  in  the  same  sense  that  a  system  with  a 
greater  value  of  application  is  more  difficult. 

The  APPL  values  that  are  input  to  the  model  are  related  to  a  standard 
schedule  of  9  months.  The  model  compares  the  schedule  for  the  project 
being  estimated  with  the  standard  schedule  and  calculates  an  effective  value 
of  APPL  which  is  used  by  ‘.he  model  to  make  estimates.  The  effective  value 
of  APPL  is  not  available  to  the  user. 

The  model  does  not  require  that  the  user  specify  the  development 
schedule.  This  is  possible,  but  the  reference  manual  recommends  that  the 
schedule  be  calculated  by  the  model  using  the  following  procedures. 

A  project  with  given  size,  application  type,  scope  of  work,  etc.  may 
be  developed  with  different  schedules.  The  effect  on  cost  of  changes  in  schedule 
is  represented  by  the  parameter  called  COMPLEXITY  (CPLX)  ".  .  .  wnicn 
provides  a  quantitative  description  of  the  relative  difficulty  of  the  design 
task."*  CPLX  describes  the  familiarity  of  the  project  staff  with  the  functions 
to  be  performed,  their  general  experience,  and  factors  that  complicate  the  develop 
ment  of  the  system  such  as  new  language,  more  than  one  user  organization, 
or  state-of-the-art  advancement.  CPLX  tends  to  be  constant  for  a  given 
organization.  It  reflects  the  way  the  organization  commits  its  resources 
in  order  to  achieve  a  perceived  proper  scheduling  of  a  project.  If  CPLX 
is  input  to  PRICE  S,  the  model  calculates  the  schedule.  If  the  schedule 
is  given,  the  model  calculates  CPLX.  If  both  are  given,  the  model  calculates 
the  schedule  according  to  the  value  of  CPLX  and  makes  adjustments  in  cost 
depending  on  whether  the  stated  schedule  represents  an  acceleration  or 
deceleration  of  the  first  schedule. 

In  any  case  the  resulting  schedule  Is  compared  with  the  9  month 
standard  schedule  to  obtain  the  effective  value  of  APPL  as  described  earl  ier 

*  Reference  Manual  PRICE  Software  Model,  RCA/PRICE  Systems,  Ch'Jrry  Hill, 

New  Jersey 
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The  initial  estimate  of  cost  in  PRICE  S  is  the  Engineering  Design 
element.  It  is  obtained  from  the  relationship: 

Engineering  Design  Cost  =  (WEIGHT)  X  (RESOURCE) 

RESOURCE  (PESO)  is  the  cost  per  pound  mentioned  above.  It  repre¬ 
sents  the  efficiency  with  which  an  organization  uses  its  resources  to 
develop  a  system.  Its  value  should  be  constant  for  an  organization.  The 
value  is  obtained  by  operating  the  model  in  a  calibration  mode  wh'ch  cal¬ 
culates  RESO  from  data  describing  past  projects. 

PESO  and  CPLX  act  together  to  describe  an  organization.  RESO  measures 
organizational  attributes  that  affect  cost  independently  of  schedule,  while 
CPLX  measures  those  attributes  that  affect  schedule. 

Other  parameters  than  the  ones  described  so  far  are  used  to  calculate 
cost.  These  will  be  described  later  under  the  description  of  inputs.  The 
primary  purpose  of  this  presentation  of  the  model  is  to  describe  the  general 
model  structure  and  the  principal  inputs.  Figure  A-1  describes  the  sequence 
of  the  calculations  and  the  model  variables. 

The  calculation  of  the  cost  elements  follows  from  the  initial  estimate 
of  Systems  Engineering  Design  Cost  using  a  sequence  of  allocations  called 
the  "Ripple  Effect."  These  are  shown  in  Table  A-3.  These  allocations  can 
be  modified  by  user  inputs.  The  cost  elements  are  defined  in  the  discussion 
of  outputs . 

Outputs 

PRICE  S  offers  a  number  of  operating  modes.  Many  of  these  modes 
involve  tailoring  or  constraining  the  development  process  to  satisfy  user 
requirements.  In  these  cases  the  normal  outputs  of  the  model  may  become 
inputs.  The  following  presentation  will  assume  that  the  standard  estimating 
situation  is  the  description  of  software  and  unconstrained  resources  resulting 
in  model  estimates  of  cost  and  schedule.  Specified  values  and  constraints 
will  be  treated  as  special  cases. 
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Sequence  jf  Calculations  in  PRICE  S 


TABLE  A- 3 


SOFTWARE  PROGRAM  COSTS  RIPPLE  EFFECT 


COSTS  IN  OOLLARS/1000 

SYSTEMS  ENGINEERING 
PROGRAMMING 
CONFIG  CONTROL.  Q/A 
DOCUMENTATION 
PROGRAM  MANAGEMENT 
TOTAL 


TOTAL 

761. 

439. 

312. 

15Z 

111. 

1775. 
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Development  Schedule 

PRICE  S  divides  the  software  development  cycle  into  three  phases: 

•  Design 

•  Implementation 

•  Test  and  Integration 

The  beginning  and  ending  month  and  year  is  given  for  each  phase  along  with 
a  bar  graph  representation  of  the  schedule.  The  phases  are  allowed  to 
overlap  in  time. 

The  Design  Phase  begins  with  the  design  of  the  system  to  be  developed 
under  the  project.  Activities  in  the  Design  Phase  include: 

•  Establish  system  architecture 

•  Allocate  system  requirements  to  programs 

•  Design  programs  in  detail 

When  each  program  design  is  completed,  coding  can  begin  in  that  program. 

If  it  is  necessary  to  change  the  program  design,  the  activity  of  design 
is  considered  part  of  the  design  cost  even  if  coding  has  begun.  AU^ough 
this  is  a  desirable  distinction,  the 'definition  of  the  programming  activity 
(see  below)  does  not  permit  the  user  to  know  how  much  redesign  cost  may 
be  included  in  an  estimate. 

The  Implementation  Phase  is  devoted  to  writing  the  program  code 
and  debugging  the  individual  programs.  Under  the  development  concept 
reflected  in  the  model,  programs  are  designed,  coded  and  debugged  as 
individual  units.  Therefore,  the  Implementation  Phase  begins  wnen  coding 
starts  on  the  first  program  to  complete  the  design  process  and  continues 
until  the  last  program  is  ready  for  formal  testing. 

Implementation  Phase  activities  include: 

•  Program  coding 

•  Program  debugging 

•  Program  documentation 
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The  Test  and  Integration  Phase  begins  with  the  test  planning  activity. 
Therefore,  it  can  start  before  any  coding  begins.  It  extends  until  the 
system  is  accepted  by  the  user.  The  major  activities  include: 

•  Test  planning 

•  System  construction  from  individual  programs 

•  Program  testing 

•  System  testing 

The  overall  development  schedule  is  obtained  from  the  user-specified 
start  date  (DSTART)  and  the  system  complexity  (CPLX).  In  the  absence  of 
any  schedule  constraints  the  model  calculates  a  nominal  schedule.  The 
calculation  of  resource  distributions  among  the  three  phases  and  the  over¬ 
lap  are  performed  using  Beta  functions. 
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The  shapes  of  the  three  profiles  can  be  changed  by  the  user.  PRICE 
S  adjusts  the  overlap  to  obtain  a  smooth  shape  of  the  total  resource  curve 
over  time. 

Development  Cost,  Constant  Dollars 

Development  Cost  is  given  by  phase  and  activity.  The  phases 
(Design,  Implementation,  and  Testing  and  Integration)  are  described  above. 
Each  phase  is  divided  into  5  activities  or  cost  elements: 

•  System  engineering 

•  Programming 

•  Configuration  control  and  quality  assurance 

•  Documentation 

•  Program  management 

System  Engineering  is  the  technical  direction  of  the  system  develop¬ 
ment.  It  includes  the  following  tasks: 

•  Development  of  system  specifications 

•  Allocation  of  the  system  functions  to  programs 

•  Description  of  program  interfaces 
e  Evaluation  of  system  performance 
e  Problem  resolution 

The  Programming  activity  includes  design  and  coding,  and  testing 
individual  programs.  These  three  tasks  are  normally  performed  in  the 
three  corresponding  phases,  but  as  was  stated  above  design  can  occur  in 
the  Implementation  Phase.  Other  activities  can  also  occur  in  different 
phases . 
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Documentation  Includes: 

•  Draft  preparation 

•  Editing 

•  Reproduction 

•  Distribution 

•  Review 

•  Revision 


Configuration  Management  is  the  control  of  the  description  of  the 
approved  system.  The  three  principal  tasks  are: 

•  Defining  the  system  baseline 

•  Managing  the  process  for  changing  the  baseline 

•  Disseminating  information  describing  the  system 

Program  Management  includes  .  .  the  supervisory,  financial,  legal, 
and  general  administrative  tasks  necessary  to  plan,  organize,  direct  and 
control  the  project."* 


The  preceding  definitions  along  with  the  definitions  of  the  develop¬ 
ment  phases  are  presented  to  give  an  idea  of  what  the  PRICE  S  creators 
consider  to  be  the  principal  cost  elements  of  a  software  development  project. 
However,  the  nominal  allocations  of  the  costs  by  element  (see  the  ripple 
calculation  in  the  Description  of  the  Model)  can  be  changed  by  the  user 
to  suit  his  own  definitions.  The  ability  to  reallocate  costs  when  exer¬ 
cised  along  with  the  calibration  of  the  model  using  cost  values  represen¬ 
tative  of  a  given  organization  make  it  possible  for  the  user  to  define 
the  model  cost  elements  almost  any  way  he  pleases. 


Costs  are  calculated  using  a  fixed  reference  year  (1976  for  the 
version  of  PRICE  S  tested)  and  corrected  for  inflation  to  the  first  of 
the  year  in  which  the  project  start  date  occurs.  An  internal  tab'’e  of 
inflation  rates  (RTABLE)  accomplished  the  adjustment  of  value.  The  stand 
ard  table  can  be  changed  by  the  user  if  desired.  Under  the  constant 
dollar  option,  the  costs  are  in  base  year  dollars. 


Op.  Cit. 
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PRICE  S  provides  an  option  for  presenting  costs  in  other  currencies 
An  input  value  establishes  the  conversion  rate  between  dollars  and  the 
other  currency. 

Development  Cost,  Inflated  Dollars 

PRICE  S  includes  an  option  that  allows  the  user  to  obtain  all  cost 
elements  in  terms  of  an  inflated  currency.  Dollars  (or  another  specified 
currency)  are  converted  from  the  base  year  to  time  during  the  development 
using  a  table  of  constants  (RTA8LE)  that  is  either  specified  by  the  user 
or  obtained  by  default.  The  effect  of  inflation  on  each  cost  element  is 
determined  by  the  Beta  function  distribution  of  resources  over  time  (see 
above) . 

Development  Effort 

An  option  is  available  under  which  PRICE  S  presents  the  cost  ele¬ 
ments  in  terms  of  effort  rather  than  currency.  The  user  can  select  either 
man-months  or  man-hours.  All  reports  are  appropriately  labeled,  ^lince 
the  internal  calculations  are  in  terms  of  dollars,  the  user  must  specify 

the  cost  per  man-month  to  effect  the  conversion  to  the  desired  output.- 

The  effects  of  inflation  on  effort  are  obtained  as  in  the  case  of 
reporting  cost  in  currency. 

Model  Output  Options 

PRICE  S  incorporates  many  execution  options  and  provides  for  many 

alternative  presentations  of  the  output  data.  The  model  allows  the  user 

a  great  amount  of  flexibility  in  stating  his  inputs  and  obta^'-rg  reports. 
It  should  be  noted,  however,  that  these  diverse  representations  need  net 
be  exercised  if  the  user  elects  to  use  the  default  values  provided  by  the 
model.  The  following  are  a  few  of  the  model  output  options. 

normal  Output 

The  default  report  written  by  PRICE  S  includes  information 
bing  the  software  and  project  parameters,  cost  and  schedule.  A  sampl-' 
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report  is  shown  in  Figure  A-2.  PRICE  S  includes  print  control  options  that 
present  specific  portions  of  the  information  in  Figure  A-2. 

Sensitivity  Option 

The  effect  on  cost  of  changing  the  values  of  four  variables  can  be 
conveniently  shown  in  a  3X3  matrix.  Under  one  option  RESOURCE  and 
COMPLEXITY  are  given  nominal,  higher  and  lower  values  and  the  development 
costs  for  the  nine  pairs  are  calculated.  Another  option  performs  the  same 
calculations  for  the  APPLICATION  and  INSTRUCTIONS  inputs.  Under  either 
option  the  user  may  let  the  model  set  the  ranges  of  the  input  values  or 
he  may  specify  them  himself.  The  options  offer  a  convenient  way  to  examine 
how  uncertainty  in  the  most  important  inputs  affects  the  estimated  develop¬ 
ment  cost.  Figure  A-3  shows  the  matrices  printed  using  the  sensitivity 
options . 

Schedule  Option 

If  the  user  specifies  the  development  schedule,  this  option  compares 
the  user's  schedule  with  the  model's  normal  schedule  and  prints  the  effect 
on  the  develo-PTiient  cost  of  departing  from  the  normal  schedule.  It  is  axio¬ 
matic  in  the  PRICE  S  model  that  either  increases  or  decreases  from  the 
normal  schedule  have  the  effect  of  increasing  the  development  cost. 

Curve  Option 

The  modal  prints  a  monthly  history  of  the  effort  and  cost.  A  cumu¬ 
lative  percent  completion  is  reported  for  each  phase.  st  is  given  as 
monthly  and  cumulative  values  and  their  related  pt  ''i  .*  -. 

Desiqn-to-Cost  Option 

Given  values  of  the  target  cost,  APPLICATION,  RESOURCE,  and  COMPLEXITY, 
PRICE  S  calculates  the  size  of  the  largest  system  that  can  be  built. 
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-  PRICE  SOFTWARE 

MODEL  — 

DATE  29-AUG-79 

TIME  09:47 

FILENAME:S3 

<790239) 

SAMPLE  SOX  1 

SAMPLE  FILE 

DESCRIPTORS 

INSTRUCTIONS  36003 

APPLICATION 

5.30:* 

RESOURCE 

3.50 

UTILIZATION  0.33 

PLATFORM 

1 .40 

COMPLEXITY 

1 .25 

NEW  DESIGN  3.5SA 

NEW  CODE 

3  .  99>* 

COSTS  IN  DOLLARS/1000 

DESIGN 

IMPL 

T  t  1 

TOTAL 

SYSTEMS  ENGINEERING 

375  . 

40  . 

244  . 

659. 

PROGRAMMING 

69. 

I9t. 

122. 

382. 

CONFIG  CONTROL,  O/A 

62. 

56  . 

149  . 

263  . 

DOCUMENTATION 

54. 

17  . 

59. 

131  . 

P.ROGRAM  MANAGEMENT 

50. 

17. 

30  . 

97  . 

TOTAL 

6  11. 

322. 

604. 

1537. 

SCHEDULE  AND  constraints 

DESIGN 

IMPL 

T  1  I 

start  work 

OCT  SO 

r£3  3  1  >* 

MAY  8  11* 

END  UORX 

JUL  SI!* 

DEC  8  1  s* 

AUG  32-* 

COST  PER  MAN-«ONTH( 1 930 

DOLLARS) 

0.0 

0.0 

0  .  0 

maximum  man-months  per 

MONTH 

0.0 

0.0 

3.0 

application  CATEGORIES 

NEW  DEVELOPMENT 

HARDWARE  INT 

ERFACES 

MIX 

DESIGN 

CODE 

TYPES  OUANTITY 

DATA  S/R  0.3  _ 

0  .  0 

0  .  0 

0 

0 

online  comm  3.03 

1  .00 

1.00 

1 

1 

REALTIME  C4C  0.C3 

1  .00 

1.00 

2 

2 

INTERACTIVE  0.23 

1  .00 

1.00 

1 

2 

MATHEMATICAL  3.23 

0.50 

0.70 

STRING  MANIP  0.26 

1.00 

1.00 

* 

•‘jJK 

OPR  SYSTEMS  0.37 

I  .00 

I  .  00 

SIZING  DATA 

FUNCTIONS  3 

STRUCTURE 

0  .  3 

LEVEL 

0  .  0 

CAPACITY  0 

SOURCE 

56961* 

EXPANSION 

6 . 32 

supplemental  information 

YEAR  1980 

MULTIPLIER 

1.300 

ESCALATION 

3.3 

TARGET  COST  0. 

INTEGRATION 

3.50 

ESC  EFFECT 

1  .  OOK 

SCHEDULE  GRA 

PH 

OCT  80 

AUG  82 

DESIGN 

IMPLEf-lENT 

test  2  INTEGRATE 


Figure  A-2.  Standard  PRICE  S  Cost  Report 
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PRICS  SOFTWARE  MODEL 


DATE  29-AUG-79 
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Figure  A-3.  Sensitivity  Analyses 


System  Integration  and  Test  Option 

This  operating  mode  calculates  the  cost  associated  with  a  system 
that  is  composed  of  independently  developed  parts.  The  cost  of  such  a 
development  is  greater  than  the  sum  of  the  costs  of  the  pieces.  Additional 
costs  are  incurred  for  defining  and  maintaining  the  specifications  of  the 
subsystems  and  their  interfaces.  There  are  also  costs  associated  with 
integrating  the  subsystems  into  the  total  system  and  conducting  total 
system  tests. 

The  amount  of  integration  and  test  cost  is  determined  by  a  single 
input  value  (INTEG)  for  each  subsystem.  The  value  of  INTEG  "...  relates 
the  level  of  engineering,  programming  and  testing  effort  involved  to 
integrate  the  subsystem  into  the  total  unified  operation."^  It  takes  on 
values  between  0  and  1  . 

The  System  Integration  and  Test  Option  is  unique  among  the  models 
tested  because  it  presents  costs  for  the  Individual  subsystems  as  well  as 
the  total  system  with  the  added  cost  of  integration. 

Verification  and  Validation  Option 

PRICE  S  calculates  the  cost  of  independent  verification  and  valida¬ 
tion  of  the  new  system  using  values  of  INTEG  between  .7  and  .8  and  proceeding 
as  above. 

Test  Bed  Option 

The  cost  of  installing  the  new  system  on  a  computer  other  than  the 
one  which  it  was  developed  is  estimated.  The  model  performs  cost  calcu¬ 
lations  based  on  the  assumption  that  installation  on  a  new  computer  involves 
redesigning  and  recoding  a  small  part  of  the  code.  Ten  percent  redesign 
and  rewrite  is  considered  representative. 


*  Op'.'  Cit. 
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Other  Options 

PRICE  S  has  many  additional  output  options  including: 

•  Condensed  cost  and  project  reports  (4  options) 

•  Subsystem  level  reports 

•  Model  constants  report 

c  Inflation  rate  table 

•  Cost  multiplier  table 

•  Effort  distribution  constants 

•  Sensitivity  constants 

•  Resource  allocation  profiles 

•  Namelist  table 

Inputs 

Software  development  projects  and  their  environments  can  be  described 
by  as  many  as  64  constants  and  4  tables  (Figure  A-4).  The  use  of  some  of 
the  inputs  excludes  some  of  the  others  and  most  of  the  parameters  have 
default  values  provided  by  the  model.  Therefore,  the  user  may  describe  a 
given  development  effort  using  different  but  equivalent  inputs  (e.g., 
number  of  Object  Instructions  or  number  of  Source  Statements  and  Expan¬ 
sion  Ratio)  and  at  different  levels  of  detail  (e.g.,  an  assumed  value  of 
RESOURCE  instead  of  a  description  of  MIX).  It  is  possible  to  execute  the 
model  by  specifying  only  8  values; 

•  INST 

•  APPL 

•  RESO 

•  UTIL 

•  PLTFM 

•  CPLX 

•  YEAR 

•  MULT 

The  following  discussion  explains  each  of  these  quantities,  and 
selected  others  that  serve  to  describe  the  model's  ability  to  define  soft¬ 
ware  and  the  development  environment. 
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The  size  of  the  software  system  produced  by  the  development  project 
is  stated  in  terms  of  INST,  .  .  the  total  number  of  delivered,  execu¬ 
table,  machine-level  instructions.  Comments,  format  statements  and  data 
declaration  statements  should,  in  general,  not  be  included."* 

Delivered  instructions  limits  the  size  measure  to  programs  that  are 
turned  over  to  the  customer.  This  would  exclude  special  development  pro¬ 
grams,  file  conversion  routines  or  test  drivers. 

Executable  instructions  are  those  that  involve  computer  operations 
in  contrast  with  data  and  constant  storage. 

Machine-level  instructions  are  the  most  elementary  operations  of 
the  computer.  Each  one  may  require  from  one  to  several  words  of  primary 
storage.  PRICE  S  operates  internally  using  the  above  definition  of  size. 
However,  the  model  offers  two  alternative  size  measures.  The  first  is  in 
terms  of  number  of  source  instructions  and  an  expansion  ratio;  the  second 
uses  the  number  of  system  functions;  and  the  third  alternative  uses  the 
system  logic  structure. 

The  selection  of  source  statements  (SOURCE)  and  expansion  ratio 
(EXPAN)  as  the  system  size  measure  offers  some  flexibility  in  the  defini¬ 
tion  of  source  program  size.  It  is  necessary  only  that  the  product  of  the 
expansion  ratio  and  the  source  program  measure  approximate  the  number  o^ 
delivered  executable  machine-level  instructions .  Therefore,  the  expansion 
ratio  can  include  an  allowance  for  comments  and  data  storage  instructions 
as  long  as  the  proportion  of  these  to  the  total  number  of  statements  is 
relatively  constant.  The  number  of  machine-level  instructions  is  e-*: 
mated  by  the  product  of  the  number  of  source  statements  and  the  expans  ,,r 
ratio; 


INST  =  SOURCE  X  EXPAN 


The  second  alternative  size  measure  uses  the  product  of  the  number 
of  functional  modules  to  be  included  in  the  software  development  (FUNCT) 
and  the  average  number  of  machine  instructions  per  function  (INSPF): 

INST  =  FUNCT  X  INSPF 


INSPF  is  a  table  entry  with  a  programmed  value  of  90.  The  user 
may  specify  a  different  value. 


The  third  method  for  specifying  size  uses  an  empirically  derived 
variable  (STRU)  to  relate  the  number  of  functional  modules  (FUNCT)  and 
the  average  functional  level  of  the  system  (LEVEL): 


1  ?  +  ^TPII 

FUNCT  =  STRU  (1  +  LEVEL) 


PRICE  S  will  calculate  values  of  STRU  given  FUNCT  and  LEVEL  froir.  past 
projects.  The  values  obtained  can  be  used  to  make  new  estimates. 

\ 

LEVEL  is  obtained  from  the  functional  tree  diagram  (Figure  A-5), 


TREE 

LEVEL 


.  PRODUCT  .10 
SU.\  7 

•  l.«9 


COMPUTATION: 


TREE 

LEVEL 

NO.  OF 

FUNCTIONAL 

MODULES 

PROQUCT 

0 

1 

0 

1 

2 

2 

2 

i 

3 

SUM 

7 

10 

Figure  A-5.  Computation  of  LEVEL 


A-51 


LEVEL  is  a  weighted  average  with  the  weights  being  the  number 
of  functions  at  each  level  of  the  tree.  LEVEL,  like  STRLI,  can  be  calculated 
from  projects  similar  to  the  one  being  estimated. 

Once  a  value  of  FUNCT  is  obtained,  INST  is  calculated  as  before  by 
using  INSPF. 

Appl ication 

PRICE  S  uses  this  parameter  to  characterize  the  difficulty  of  the 
programming  task.  It  is  intended  to  adjust  the  relationship  between  cost 
and  program  size  to  account  for  the  inherent  differences  in  resources 
associated  with  different  types  of  applications.  The  application  param¬ 
eter  (APPL)  .  .  represents  an  inherent  instruction  complexity,  inde¬ 
pendent  of  variation  in  resources,  schedules,  operating  environnvent  and 
system  utilization."* 

Acceptable  values  of  APPL  range  from  0.866  to  10.952.  The  lower 
end  of  the  range  is  associated  with  pregrams  that  are  predominantly  math 
and  string  manipulations;  the  higher  values  represent  real-time  command 
and  control  and  interactive  applications.  Increasing  values  of  APPL 
describe  programming  tasks  that  require  mors  resources  for  a  system  of 
given  size. 

Values  of  APPL  for  a  given  estimating  situation  may  be  assigned 
on  the  basis  of  experience  with  .similar  systems.  Or,  APPL  for  the  system, 
may  be  calculated  from  a  weighted  sum  of  its  component  parts.  This  alter¬ 
native  determination  of  APPL  is  obtained  by  estimating  the  proportion  of 
the  total  system  size  represented  by  each  of  seven  categories  of  automated 
functions  (Mix  Categories,  Table A-4).  The  model  lets  the  user  define  his 
own  category  and  APPL  value  if  necessary. 

C  i  t . 


TABLE  A- 4 

MIX  Categories 


MIX  CATEGORY 

■ ■  ■—  -  -  1 

IDENTIFYING  CHARACTERISTICS  j 

DATA  STORAGE  AND  RETRIEVAL: 

-OPERATION  OF  DATA  STORAGE  DEVICES 

(MOAT) 

-DATA  3ASE  .MANAGEMENT  | 

-SECONDARY  STORAGE  HANDLING 

APPL>4 . 1 0 

-DATA  SLOCKING  AND  OE8LOCKI.NG  ! 
-HASHING  TECHNIQUES  i 
-HARDWARE  ORIENTED  ' 

ON-LINE  COMMUNICATIONS: 

-MACHINE-TO-MACHINE  COMMUNICATIONS  ; 

(4MONL) 

WITH  QUEUING  PERMITTED.  1 

-TIMING  REQUIREMENTS  NOT  AS 

APPL-6 .  t  6 

_ 

RESTRICTIVE  AS  WITH  REAL  TIME  1 

COMMAND  AND  CONTROL  1 

1 

REAL  TIME  COMMAND  AND 

CONTROL: 

-MACHINE-TO-MACHINE  COMMUNICATIONS  j 

(MREA) 

UMDER  TIGHT  TIMING  CONSTRAINTS  1 

-QUEUING  NOT  PRACTICABLE 

AP9L«8 . 46 

-HEAVY  HARDWARE  INTERFACE  i 

-STRICT  PROTOCOL  REQUIRE.MENTS 

INTERACTIVE  OPERATIONS: 

-HAN-MACHINE  INTERFACES 

( MINT) 

-HUMAN  ENGINEERING  CONSIDERATIONS 

-ERROR  DETECTION  AND  PROTECTION 

APPL« 10.95 

.MATHE.MATICAL  APPLICATIONS: 

-ROUTINE  MATHEMATICAL  APPLICATIONS 

(MKAT) 

WITH  NO  OVERRIDING  CONSTRAINTS 

APPL-0 . 36 

STRING  .MANIPULATION: 

-ROUTINE  APPLICATIONS  WITH  NO 

(MSTS) 

OVERRIDING  CONSTRAINTS 
-NOT  ORIENTED  TOWARD  MATHEMATICS 

APPL-2 . 3 ' 

-TYPIFIED  3Y  LANGUAGE  COMPILERS, 
SORTING,  FORMATTING,  SUFFER 
MANIPULATION,  ETC. 

OPERATING  SYSTEMS! 

-TASK  MANAGEMENT 

f MOPR) 

-MEMORY  MANAGEMENT 
-HEAVY  HARDWARE  INTERFACE 

APPL- 10.95 

-STRICT  TIMING  REQUIREMENTS 
-HIGH  RELIABILITY 
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Given  the  proportions  of  code  in  each  category  the  system  applica¬ 
tion  is  calculated  as  follows: 

a 

APPL  »  )  (MIX  )(APPL.) 

-j  1 

i»l 

where  MIX^  =  the  proportion  of  the  system  code  in  the  ith  WIX  category 
APPL^.  *  the  APPL  value  for  the  ith  category  (see  Table  A- 4). 

Resource 

RESO  represents  the  effects  on  cost  of  items  such  as:".  .  .  sKill 
levels  experience,  productivity,  efficiency,  computer  operating  charges, 
and  labor  and  overhead  rates  of  the  organization."*  The  PRICE  S  estimating 
procedure  reflects  the  assumption  that  this  value  remains  fixed  in  an 
organization.  The  value  of  RESO  is  obtained  from  historical  data  using 
the  PRICE  S  calibration  mode. 


A  large  organization  that  includes  many  separate  groups  may  present 
different  values  of  RESO.  In  making  cost  estimates  for  such  organizations 
it  would  be  necessary  to  ascertain  that  the  RESO  value  is  consistent  with 
the  particular  group  that  will  undertake  the  project  being  estimated. 

Utilization 

UTIL  describes  the  proportion  of  available  computer  memory  occupied 
by  the  application  programs.  It  also  describes  the  fraction  of  the  computer 
cycle  time  required  to  execute  the  program. 


The  PRICE  S  Reference  Manual  does  not  describe  how  combined  time 
and  space  constraints  are  represented  by  UTIL.  Discussions  with  PRICE  S 
staff  members  suggest  that  the  parameter  represents  a  subjective  assess¬ 
ment  of  the  effect  of  either  one  or  both  types  of  constraint  in  a  given 
situation. 


A- 54 


There  is  no  effect  on  cost  associated  with  values  of  UTIL  less  than 
0.5,  while  values  of  UTIL  greater  than  0.9  have  a  very  large  effect  (see 
Figure  A-6) . 

Values  of  UTIL  for  shipborne  or  mobile  applications  range  from  0.65 
to  0.75,  airborne  applications  range  from  0.75  to  0.85,  and  space  systems 
have  values  close  0.9. 

PI atform 

PLTFM  "...  denotes  the  operating  environment  of  the  software, 
and  is  a  measure  of  portability,  reliability,  structuring,  and  test  and 
documentation  requirements  to  be  provided  for  acceptable  contract  perfor¬ 
mance.  "  * 


PLTFM  describes  the  specifications  to  be  satisfied  by  the  software. 
It  represents  the  degree  of  testing  and  documentation  associated  with  it. 
The  lower  values  denote  one-time,  in-house  software  with  little  or  no 
documentation.  Increasing  values  describe  more  stringent  testing  and 
documentation  up  to  systems  such  as  man-rated  space  applications.  Table  A 
shows  typical  values  of  PLTFM. 

Complexity 

CPLX  quantifies  the  effect  on  the  time  required  to  complete  the 
software  development  of  the  organization's  readiness  to  undertake  the 
project. 

Values  of  CPLX  range  from  -0.2  to  +0.6.  Increasingly  negative 
values  describe  projects  undertaken  by  experienced  groups  working  on  appl  ’ 
cations  very  similar  to  ones  that  have  been  done  before.  The  larger  posi¬ 
tive  values  would  represent  projects  in  which  there  are  one  or  more  ta  ‘r- 
that  have  been  associated  with  longer  development  times.  Such  factors 
include  inexperienced  crews,  unfamiliar  applications,  new  hardware  or 
software  and  so  forth.  Table  A-6  shows  the  values  of  CPLX  associated 
the  existence  of  different  personnel  and  environmental  conditions. 


COST 


TABLE  A-5 


TYPICAL  PLTFM  VALUES 


OPERATING  ENVIRONMENT 

PLTFM 

■~1 

1 

PRODUCTION  CENTER  -  INTERNALLY  DEVELOPED  SDFTWARE 

0.6-0. 8 

PRODUCTION  CENTER  -  CONTRACTED  SOFTWARE 

1.0 

MIL-SPEC  GROUND 

1 .2 

MILITARY  MOBILE  (VAN  OR  SHIPBOARD) 

: 

1.4 

COMMERCIAL  AVIONICS 

1.7 

1 

MIL-SPEC  AVIONICS 

1.8 

UNMANNED  SPACE 

2.0 

MANNED  SPACE 

2.5 

—1 
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TYPICAL  CPLX  VALUES 
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ADV/'NCEMENT 


PRICE  S  considers  CPLX  and  schedule  to  be  alternative  representations 
of  the  time  required  to  complete  a  software  development  project.  If  either 
one  is  given,  the  model  can  compute  the  other. 

Values  of  CPLX,  as  in  the  case  of  PESO,  tend  to  be  constant  for  a 
given  organization.  The  calibration  mode  is  used  to  obtain  values  from 
representative  past  projects.  These  are  used  to  make  estimates  for  new 
software  development  efforts. 

Reference  Year 

PRICE  S  incorporates  a  reference  calendar  that  is  used  to  calculate 
changes  in  the  value  of  the  monetary  unit  and  the  rate  of  technological 
change. 


The  reference  year  for  PRICE  S  cost  calculations  is  1976.  The 
inflation  rate  table,  RTABLE,  is  used  to  adjust  calculated  costs  to  the 
year  specified  by  YEAR, 

If  no  project  start  date  is  given,  the  model  assumes  a  start  date 
of  1  January  of  YEAR. 

YEAR  may  be  used  to  define  the  state-of-the  art  of  system  develop¬ 
ment  technology.  It  may  be  used  along  with  the  input  TECIMP  to  represent 
the  decrease  in  cost  associated  with  expected  improvements  in  development 
efficiency.  TECIMP  is  included  in  OTABLE  and  represents  the  difference 
in  development  technology  expected  to  occur  in  the  interval  between  YEAR 
and  the  start  of  the  project,  OSTART. 

Cost  Multiplier 

MULT  is  a  multiplier  for  all  cost  values.  "Its  primary  purpose  is 
to  adjust  all  costs  to  include  mark-ups,  such  as  6&A,  IRiD,  and  profit 
or  fee."* 


*  Op.  Cit. 
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Other  Inputs 

PRICE  S  includes  many  Inputs  in  addition  to  the  required  ones 
described  above.  These  optional  inputs  serve  to  define  the  system  in 
greater  detail  when  desirable  and  can  specify  constraints  on  the  project 
development  parameters.  They  include: 

NEWD  -  The  amount  of  new  design  required  for  the  software  develop¬ 
ment  (Range:  0,1) 

NEWC  -  The  amount  of  new  code  required  for  the  software  development 
(Range:  0,  1) 

NOTE:  NEWD  and  NEWC  are  required  Inputs  if  APPL  is  entered  rather  than 
calculated  from  the  MIX  categories. 

SCHEDULE; 

DSTART  -  The  date  design  effort  starts 

DEND  -  The  date  design  effort  ends 

ISTART  -  The  date  implementation  effort  starts 

lEND  -  The  date  implementation  effort  ends 

TSTART  -  The  date  test  and  integration  effort  starts 

TEND  -  The  date  test  and  integration  effort  ends 

RESOURCE  CONSTRAINTS; 

DCOST  -  Average  Cost  per  Man-Month/Hour  -  Design  Phase 

DMAX  -  Maximum  Man-Month/Hours  per  Month  -  Design  Phase 

ICOST  -  Average  Cost  per  Man-Month/Hour  -  Implementation  Phase 

IMAX  -  Maximum  Man-Month/Hours  per  Month  -  Implementation  Phase 

TCOST  -  Average  Cost  per  Man-Month/Hour  -  Test/Integration  Phase 

TMAX  -  Maximum  Man-Month/Hours  per  Month  -  Test/Integration 

Phase 


NEW  DESIGN;  The  proportion  of  new  design  in  each  mix  category  required 
for  the  software  development. 

DDAT  -  Date  storage  and  retrieval 

DONL  -  On-Line  communications 

DREA  -  Real-time  command  and  control 


DINT  -  Interactive  operations 

DMAT  -  Mathematical  applications 

DSTR  -  String  manipulation 

DOPR  -  Operating  systems 

OPTIONAL 

DAPP8  -  Applies  only  when  MAPP8  and  APPL8  are  specified 

NEW  CODE:  The  proportion  of  new  code  in  each  mix  category  required  for 
the  new  development. 

CDAT  -  Data  storage  and  retrieval 

CONL  -  On-Line  communications 

CREA  -  Real-time  command  and  control 
CINT  -  Interactive  operations 

CMAT  -  Mathematical  applications 

CSTR  -  String  manipulation 

COPR  -  Operating  systems 

OPTIONAL 

CAPP8  -  Applies  only  when  MAPP8  and  APPL8  are  specified 
INTERFACE  TYPES: 

TDAT  -  Data  storage  and  retrieval  devices 

TONL  -  On-Line  communications  devices 

TREA  -  Real-time  command  and  control  devices 

TINT  -  Interactive  devices 

INTERFACE  QUANTITIES: 

QDAT  -  Number  of  data  storage  and  retrieval  devices 
QONL  -  Number  on  Of-Line  communication  devices 

QREA  -  Number  of  real-time  command  and  control  devices 

QINT  -  Number  of  interactive  devices 
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CAP  -  Available  memory  size.  'JTIL  =  INST/CAP. 

INTEG  -  Adjustment  for  system  integration  cost. 

Calibration  (ECIRP) 

PRICE  S  incorporates  the  assertion  that  many  different  development 
projects  histories  can  be  associated  with  software  represented  by  a  given 
set  of  characteristics.  These  differences  can  be  attributed  to  how  an 
organization  undertakes  a  project.  PRICE  S  recognizes  two  types  of 
project  development.  An  organization  may  operate  in  a  manner  emphasizing 
tight  schedules  and  higher  staffing  rates  or  it  may  elect  to  limit  staffing 
and  extend  the  completion  time. 

The  discussion  of  inputs  described  the  parameter  PESO  to  be  asso¬ 
ciated  with  the  expenditure  of  project  resources  and  CPLX  with  the  project 
schedule.  ECIRP  is  an  execution  mode  of  PRICE  S  that  uses  historical 
project  data  to  calculate  values  of  these  parameters.  The  values  obtained 
indicate  how  the  organization  has  historically  staffed  and  scheduled  its 
projects.  The  prerequisite  for  obtaining  reliable  cost  estimates  with 
PRICE  S  is  to  verify  that  the  values  of  RESO  and  CPLX  behave  in  a  consis¬ 
tent  pattern  for  the  organization.  Establishing  these  values  using  past 
projects  provides  two  important  parameters  that  cannot  be  obtained  with 
any  confidence  from  any  outside  source. 

Data  obtained  from  an  organization's  records  reflects  all  the 
definitions  and  processes  peculiar  to  that  organization.  These  become 
implicitly  represented  in  the  calibration  parameters  and  are  reflected  in 
subsequent  estimates. 

The  ECIRP  mode  is  executed  by  entering  a  non-zero  value  for  the 
input  TARCST.  TARCST  is  the  total  development  cost  for  a  completed  project. 
Using  this  value,  the  project  schedule  descriptions  of  system  size,  appli¬ 
cation  and  other  characteristics  allows  the  mode’  to  calculate  values  for 
RESO  and  CPLX.  Repeating  the  Calibration  for  several  projects  provides 
the  basis  for  estimating  RESO  and  CPLX  for  new  projects. 


A-62 


F/S  9/2 


AD-A104  226 


UNCLASSIFIED 


GENERAL  RESEARCH  CORP  HUNTSVILLE  AL 
AN  EVALUATION  OF  SOFTMARE  COST  ESTIMATING  MODELS. (U) 

JUN  61  R  THIBODEAU  F30602-79-C-02lf4 

GRC-CR-1-940  RADC-TR-Bl-144  NL 


SLIM 

Description  of  the  Model 

SLIM  (Software  Life  Cycle  Model)  is  a  proprietary  software  cost  esti¬ 
mating  model  offered  by  Quantitative  Software  Management,  Inc.,  McLean, 
Virginia.  The  model  is  presently  resident  on  the  American  Management 
Systems  time  sharing  network  which  provides  local  dial-up  facilities  for 
a  variety  of  low-speed  terminals  in  over  200  cities  nation-wide. 

SLIM  has  its  origins  in  the  work  done  by  L.  H.  Putnam  at  the  'JS  Army 
Computer  Systems  Command.  Putnam  applied  the  hypothesis  presented  by  Norcer 
of  IBM*  that  given  linear  learning,  the  rate  of  expending  effort  on 
the  solution  of  problems  follows  a  Rayleigh  distribution  function  over  time. 
Both  Norden  and  Putnam  obtained  good  results  by  fitting  the  Rayleigh  function 
to  the  distribution  of  effort  per  unit  time  over  the  life  cycle  of  large 
software  development  projects.  The  Rayleigh  representation  was  found  to  be 
applicable  for  both  the  total  life  cycle  effort  and  for  the  component  activ¬ 
ities  such  as  Design  and  Coding,  and  Test  and  Validation. 


SYSTEMS 

OEFINI-  FUNCTIONAL 


Using  the  Rayleigh  distribution  has  the  advantage  that  the  time 
distribution  of  effort  during  the  software  life  cycle  is  determined  by 
two  parameters:  the  area  under  the  Rayleigh  curve,  which  when  applied 


P.  V,  Norden,  Useful  Tools  for  Project  Management,  Management  of 
Production,  M.  K.  Starr,  Ed.,  ^-'gu’n  Books,  1970,  pp  71-101 . 
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to  the  rate  of  expending  effort  becomes  the  total  life  cycle  effort;  and 
the  time  to  reach  the  peak  of  the  effort  rate.  Putnam  showed  that  for 
large  systems  this  time  is  the  development  time.  The  result  of  applying 
the  Rayleigh  form  to  the  software  development  cycle  is  the  equation: 


=  ^  2t  exp  (-^-2) 
t/  2tt 


where: 

y  =  the  rate  of  expending  effort,  e.g.,  man-months  per  month; 
K  =  the  total  life  cycle  effort,  man-months  or  man-years; 
tjj|  =  the  development  time,  months  or  years; 
t  =  the  time  from  the  start  of  development,  months  or  years. 


2 

Putnam  then  observed  that  the  variable  K/t .  was  correlated  with 

d 

the  subjective  difficulty  of  a  system.  He  calls  the  ratio  "Difficulty" 
and  has  determined  empirically  that  for  large  systems  (more  than  70000 
source  statements),  the  productivity  (source  statements  per  man-year  of 
development)  is  related  to  the  Difficulty  by  the  following  equation: 

W  *  CnO”^"'^ 

where: 

^  »  the  average  productivity,  source  statements  per  man-year  of 
development ; 

0  =  Difficulty,  K/t^^; 

Cn  =  empirical  constant. 


The  portion  of  the  life  cycle  that  produces  the  code  (Design  and 
Coding  Phase)  defines  productivity  and  this  activity  constitutes  1/6  the 
tota'  life  cycle  effort.  The  Design  and  Coding  Phase  begins  at  the  start 
of  the  life  cycle  and  is  approximately  95  percent  complete  at  the  end  of 
development  (some  design  and  coding  occurs  after  system  installation). 
These  observed  relationships  along  with  the  preceding  ones  define  the 
software  equation: 
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where : 


C|^  ■  the  Technology  Constant. 

Putnam  states  that  the  Technology  Constant  is  quantized  and  it 
"seems  to  relate  machine  throughput  (or  programmer  turn-around,  available 
test  time,  etc.)  and  other  technology  improvements  like  the  chief  pro¬ 
grammer  team,  top  down  structured  programming,  on-line  interactive  job 
submission,  etc."* 


The  final  equation  developed  by  Putnam  concerns  an  observed  rela- 
tionahip  between  the  type  of  development  being  undertaken,  the  development 
time  and  the  difficulty.  Putnam  shows  that  the  gradient  of  Difficulty: 


is  related  to  the  type  of  development  (e.g.,  stand-alone  system,  rebuild, 
composite,  etc.).  The  Difficulty  gradient  takes  on  a  specific  value  for 
each  type  of  system  and  has  the  effect  of  imposing  a  minimum  development 
time  for  any  system  with  a  given  Difficulty  and  total  life  cycle  effort. 
Trying  to  develop  the  system  in  a  shorter  time  increases  the  Difficulty; 
conversely,  increasing  the  development  time  decreases  the  Difficulty. 


As  a  consequence  of  the  Rayl eigh/Norden  distribution  of  effort  per 
unit  time  and  the  empirical  relationships  between  productivity  and  D  and 
between  7D  and  type  of  development,  Putnam  proposes  that  the  following 
equations  govern  the  life  cycle  effort  for  large  software  systems: 

S,  .  C,k’/3 


7D  •  ^  3 

where: 

C|^  «  the  Technology  Constant  and  is  known  for  a  given  environment; 

vO  ■  the  Difficulty  gradient  and  is  known  for  a  given  type  of 
development. 


L.  H.  Putnam,  Measurement  Oat’  tc  Sizing,  Estimatina  and  Control 

of  the  Software  Life  Cycle,  .tZE  COMPr.'v  78,  San  Francisco, 'Cal  if , 

Mar  1978,  p  12. 
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For  a  given  value  of  the  Technology  Constant,  and  type  of  develop¬ 
ment,  t^e  software  equation  and  the  Difficulty  gradient  define  time-effort- 
Difficulty  crade-offs  for  any  system  of  a  given  size. 


The  method  to  this  point  provides  a  minimum  development  time  and 

a  relationship  between  effort  and  development  time  for  a  given  system  and 

environment.  Putnam  recommends ^ ^ ^  that  the  selected  development  time 

within  the  trade-off  region  be  as  long  as  possible  within  the  constraints 

imposed  by  the  need  for  the  system.  As  can  be  seen  from  the  chart  above,  it 

’S  advantageous  to  extend  the  development  time  as  much  as  possible.  Eut  in 

a  practical  situation  there  are  several  additional  constraints  that  affect  the 

development  time  and  effort  for  a  software  system.  Putnam  has  incorporatea 

these  into  a  linear  programming  problem  which  is  cne  of  the  operating  modes 

of  SLIM.  The  linear  programming  solution  satisfies  the  following  rela- 

(2) 

t’onsmps^  . 


L.  H.  Putnam,  The  Real  Economics  of  Software  Development,  Quantitative 
Software  Management,  Inc.,  Jun  1980,  p  5. 

f  21 

^  '  L.  H.  Putnam,  Software  Costing  and  Life  Cycle  Control,  Workshop  on 
Quantitative  Software  Models,  iEEE  Cat.  No.  TH0067-9,  Oct  1979,  p  29. 
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Sj  ■  C|^  software  equation 

K/t .  <\ey_.„  maximum  peak  manpower 

K/tj  minimum  peak  manpower 

K/tj^  ^  lOl  maximum  difficulty 

0 

K/tj  1  1^0 1  maximum  difficulty  gradient 

td  ^contract  delivery  time 
J/My  (YK)  ^  total  budgeted  amount  for  development 

Providing  that  a  solution  exists  that  satisfies  all  the  constraints, 
the  result  is  a  range  of  development  times  and  efforts  extending  from  a 
minimum  time  solution  on  one  extreme  to  a  minimum  effort  or  cost  on  the 
other. 


TIME 

MAN-MONTHS 

COST  (X  $1000) 

MINIMUM  TIME 

21,93 

399 

1662 

22.43 

364 

1519 

22.93 

334 

1390 

23.43 

306 

1276 

MINIMUM  COST 

24.00 

278 

1159 

The  notion  that  the  cost  of  software  development  should  be  dependent 
on  management  strategy  as  an  explicit  input  to  the  cost  estimating  model 
is  unique.  Although  other  models  may  be  used  to  the  same  end,  SLIM  in¬ 
cludes  it  as  part  of  the  model  structure. 

Another  important  feature  of  SLIM  is  the  representation  of  the 
uncertainty  in  the  primary  estimating  variable  and  its  effect  on  the 
development  time  and  effort.  Program  size  can  be  represented  as  a  range 
of  possible  values  for  system-level  estimates  or  as  smallest,  largest, 
and  most  likely  values  for  component  estimates.  In  either  case,  an 
expected  value  and  variance  is  established  for  each  size  measure  and 


these  are  used  to  perform  a  Monte  Carlo  type  sirulation  of  the  minimum 
development  time  and  its  related  effort.  The  result  is  statistical 
distributions  of  times  and  efforts  that  can  be  used  to  establish  time  and 
effort  values  associated  with  different  values  of  risk.  The  model  uses 
these  relationships  in  several  of  its  operating  modes.  The  applications 
of  these  relationships  are  described  in  the  discussion  of  the  model  outputs. 

Having  established  the  basic  life  cycle  cost  estimates  for  a  given 
development  situation,  the  model  calculates  "front  end"  effort  and  time 
as  a  function  of  the  development  effort.  It  also  calculates  computer 
resource  requirements  and  documentation  size. 

The  description  of  SLIM  presented  here  is  based  on  materials  thac 
have  been  published  widely  by  Putnam  and  that  are  derived  in  large 
part  from  his  work  at  the  Army  Computer  Systems  Command.  The  methods 
are  applicable  to  systems  larger  than  70000  source  statements.  The 
extension  of  the  model  to  systems  between  10000  and  70000  statements  is 
held  to  be  proprietary  and  there  is  no. available  description  of  the 
method  used  to  make  estimates  for  systems  in  this  range  of  the  size 
variable.  The  evaluated  model  rejects  systems  with  fewer  than  10000 
statements.  A  new  version  of  SLIM  is  now  available  that  accepts  systems 
as  small  as  5000  statements. 

Outputs 

Development  Time 

The  development  cycle  begins  with  the  detailed  design  of  the  system 
programs.  It  is  assumed  that  the  system  requirements  and  the  system  spe¬ 
cifications  are  completed  in  activities  that  precede  the  development  cycle. 
Development  ends  when  the  system  completes  its  acceptance  testing  and  is 
released  to  the  user.  The  Development  Time  is  the  elapsed  time  in  months 
extending  from  the  start  of  program  design  until  the  system  is  released. 

Development  Effort 

All  personnel  hours,  both  direct  and  indirect,  expended  during  the 
period  defined  by  the  Development  “-re  included  in  the  development 
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effort.  In  many  models  the  precise  definition  of  the  indirect  hours  is 
an  important  factor  that  affects  the  accuracy  of  the  predictions.  The 
effects  of  variations  in  the  definitions  are  minimized  in  SLIM  (as  in  any 
models  that  are  self-calibrating)  because  the  important  model  parameters 
are  calculated  from  historical  data  that  should  reflect  the  same  cost 
accounting  practices  as  will  be  applied  to  the  project  being  estimated. 

In  using  such  models  it  is  important  that  the  historical  data  be  represen¬ 
tative  of  the  estimated  project  in  such  definitions  as  types  of  personnel 
hours  to  be  recorded  and  reported  against  the  project,  the  level  of  manage¬ 
ment  reported  directly  and  indirectly  and  the  method  of  distributing  the 
overhead  charges. 

'"he  indirect  effort  is  incorporated  into  the  SLIM  estimates  by  way 
of  the  Technology  Factor.  The  Technology  Factor  is  obtai-'ec  og  tne 
calibration  mode  of  the  model  from  cata  describing  pas:  projects. 

The  Technology  Factor  includes  a  factor  related  to  the  organizational 
productivity.  The  average  procuct: vity  is  defined  as  the  ratio  of  the  total 
end  product  code  to  the  total  effort  to  produce  the  code.  The  total 
effort  includes  all  direct  and  indirect  activities.*  Therefore,  as  long 
as  the  size  of  the  system  from  historical  records  and  the  total 
effort  to  produce  the  system  are  measured  on  the  same  basis  as  the  project 
being  estimated,  the  Technology  Factor  obtained  from  the  historical  data  will 
be  appropriate  and  the  details  of  the  definitions  of  total  effort  need  not 
be  specified  as  part  of  the  model  definitions. 

Development  Cost,  Constant  Dollars 

The  Development  Cost  is  obtained  from  the  Development  Effort  using 
a  constant  cost  per  unit  effort  (labor  rate)  measured  in  dollars  per  man 
year  for  a  specified  year.  Care  must  be  exercised  to  ensure  that  the  in¬ 
direct  and  overhead  costs  are  consistent  with  the  definitions  used  to  obtain 
the  Technology  Factor.  If  they  are  not,  omissions  or  double-counting  will 
occur . 

★ 

L.  H.  Putnam,  A  General  Empirical  Solution  to  the  Macro  Software  Sizing 

and  Estimating  Problem,  IEEE  ■" >"50530:1005  on  Software  Engineering, 

Vol  .  SE-4,  No.  4,  Jul  1978.  p  '-SS. 
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Development  Cost,  Inflated  Dollars 

Costs  are  expressed  in  current  year  dollars  using  an  inflation  rate 
specified  in  the  model  inputs. 

MODEL  OUTPUT  OPTIONS 
Simulation  Option 

This  is  the  primary  operative  mode  of  SLIM.  It  performs  the  calcu¬ 
lations  that  are  reflected  in  almost  all  the  other  options.  The  mean  and 
standard  deviation  are  estimated  for  the  following: 

•  System  Size, 

•  Minimum  Development  Time, 

•  Development  Effort, 

•  Development  Cost  (inflated  and  uninflated  dollars). 

A  sensitivity  profile  is  presented  that  shows  the  change  in  Develop¬ 
ment  Time,  Effort  and  Cost  with  System  Size. 

A  consistency  check  is  made  with  similar  systems  as  represented  by 
the  RAOC  data  base.  Calculated  values  of  development  effort,  time,  and 
productivity  and  average  number  of  personnel  are  compared  with  the  90  percent 
range  of  values  taken  from  the  data  base  for  the  given  system  size.  The 
model  indicates  whether  the  estimated  values  are  within  the  range,  above 
it,  or  below  it. 

The  variance  in  the  estimates  comes  from  the  assumed  uncertainty 
in  the  size  estimate,  the  cost  per  unit  effort,  and  the  difficulty  gradient. 


Linear  Program  Option 

The  user  specifies  the  constraints  operating  on  the  system  develop¬ 
ment  project  and  the  Linear  Program  Option  calculates  various  time  and 
effort  alternatives  that  satisfy  the  constraints.  The  constraints  include: 
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•  Maximum  Development  Cost, 

•  Maximum  Development  Time, 

•  Maximum  Number  of  People  at  Peak  Staffing, 
e  Minimum  Number  of  People  at  Peak  Staffing. 

The  software  equation  is  satisfied  subject  to  the  above  constraints. 
The  model  indicates  if  no  solution  exists.  If  a  solution  is  possible, 

SLIM  prints  a  matrix  showing  the  time,  effort,  and  cost  associated  with  a 
minimum  cost  project  and  a  minimum  time  project.  Since  these  two  solutions 
represent  only  the  extremes  of  the  feasible  region,  the  model  also  presents 
effort  and  cost  for  a  number  of  intermediate  development  times. 

Front-End  Option 

Low,  expected,  and  high  values  of  time  and  effort  are  calculated 
for  the  activities  that  precede  the  Development  Phase  of  the  software 
life  cycle.  These  include: 

•  Feasibil ity  Study, , 

•  Functional  Design. 

The  estimates  are  said  to  be  made  using  IBM  data*  and  the  values 
estimated  for  the  Development  Cycle. 

Life  Cycle  Option 

SLIM  calculates  monthly,  quarterly,  or  yearly  estimates  of  the  number 
of  people,  the  cost,  and  the  cumulative  cost  for  the  system  life  cycle. 

The  mean  and  standard  deviation  is  presented  for  each  value. 

Risk  Analysis  Option 

Tables  are  oresented  that  indicate  the  probability  that  time,  effort, 
and  cost  to  develop  the  system  wiVl  not  exceed  the  given  amounts.  From 
these  outputs  it  is  possible  to  make  an  inference  such  as:  "There  is  a 
95  percent  probability  given  the  input  assumptions  that  the  system  develop¬ 
ment  will  not  take  longer  than  22.3  months."  Similar  statements  can  be 
made  for  effort  and  cost. 

SLIM  User's  Guide,  Quantita:  Sc'*:.'.  ■  e  ‘-"araGement ,  Inc. 
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Benefit  Analysis  Option 

Using  the  user-specified  economic  life  of  the  system  and  the  desired 
annual  v*ate  of  return,  the  model  calculates  the  discounted  cash  flow  value 
of  the  system  that  amortizes  its  development  and  maintenance  cost. 

New  Schedul e  Option 

The  Simulation  option  provides  estimates  of  time,  cost,  and  effort 
based  on  the  minimum  time  to  develop  a  given  system.  The  New  Schedule 
option  permits  the  user  to  specify  times  greater  than  the  minimum  develop¬ 
ment  time  and  obtain  corresponding  estimates  of  effort  and  cost.  The 
results  are  compared  with  similar  size  system  experience  as  in  the  case 
for  the  Simulation  option.  The  Manloading,  Cashflow  and  Life  Cycle  options 
may  be  executed  and  reflect  the  new  schedule. 

Design  to  Cost  Option 

Given  an  effort  less  than  that  established  by  the  minimum  time 
(maximum  effort)  solution  will  result  in  a  new  development  time  and  cost. 
Consistency  checks  and  subsequent  executions  of  the  Manloading,  Cashflow, 
and  Life  Cycle  options  are  executed  in  the  same  manner  as  the  New  Schedule 
option. 

Design  to  Risk  Option 

The  user  is  asked  to  choose  among  three  levels  of  risk  (.99,  .95, 
and  .90)  describing  the  probability  of  exceeding  a  user-specified  maximum 
development  time.  The  model  calculates  expected  values  and  standard  devia¬ 
tions  of  development  time,  effort,  and  cost  associated  with  the  selected 
level  of  risk.  The  Manloading,  Cashflow,  and  Life  Cycle  options  may  be 
executed  using  the  new  parameters. 

Other  Options 

SLIM  has  several  additional  output  options  including: 
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Manloading  by  Project  Month 
Cashflow  by  Project  Month 
Major  Milestones 
Monthly  Code  Production 
Monthly  Computer  Hours 
Documentation 
PERT  Sizing 


Inputs 

SLIM  has  three  primary  inputs:  System  Size  in  number  of  developed 
source  statements,  Level  or  Difficulty  Gradient,  and  Technology  Factor. 

The  System  Size  is  estimated  by  someone  familiar  with  the  functions  to  be 
automated  and  is  the  primary  descriptor  of  the  work  to  be  done.  The  other 
two  inputs  affect  the  type  of  effort  involved  and  the  development  environ¬ 
ment.  These  are  obtained  from  user  experience  with  previous  SLIM  estimates 
and  by  calibrating  the  model  using  historical  data  that  is  representative 
of  the  project  to  be  estimated.  The  process  of  obtaining  these  values  is 
described  below  in  the  section  on  calibration. 

Additional  input  parameters  describe  other  properties  of  the  system 
and  the  development  process. 

System  Size 

The  system  size  is  described  in  terms  of  the  number  of  executable 
source  language  statements  to  be  written.  Data  declaration  and  input/ 
output  statements  are  included  in  the  size  measure,  but  comment  statements 
are  not.  SLIM  accepts  two  alternate  methods  for  describing  system  size. 

The  first  is  suggested  for  use  during  the  early  phases  of  the  life  cycle 
before  the  system  functions  are  defined.  The  user  makes  an  estimate  of  the 
the  possible  range  of  values  for  the  total  system  size.  The  model  uses 
this  range  to  calculate  the  expected  value  of  the  system  size  and  its 
standard  deviation. 
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The  representation  of  the  system  size  as  a  random  variable  is  used 
to  calculate  the  effect  of  the  uncertainty  in  the  size  measure  on  the  model 
estimates.  Using  the  Monte  Carlo  technique,  SLIM  performs  repeated  calcu¬ 
lations  of  the  output  values  using  values  of  the  system  size  defined  by  a 
normal  distribution  with  the  mean  and  standard  deviation  calculated  as 
described  above.  The  statistics  of  the  outputs  (mean  and  standard  devia¬ 
tion)  are  printed  and  these  values  are  used  in  executing  several  of  the 
model  options.  Ultimately  this  approach  gives  the  user  an  explicit  state¬ 
ment  of  the  risk  assiciated  with  the  model  predictions. 

The  second  method  used  by  SLIM  to  calculate  the  system  size  requires 
the  user  to  make  estimates  of  the  sizes  of  each  of  the  system  functions. 

It  is  suggested  that  several  analysts  make  three  estimates  of  the  size  of 
each  system  function:  the  least  possible  number  of  statements,  the  greatest, 
and  the  most  likely.  These  are  used  to  make  composite  estimates  for  each 
function  by  combining  the  range  values  and  calculating  the  mean  of  the 
expected  values  for  each  function.  The  three  values  obtained  for  each 
function  are  used  to  calculate  the  system  expected  value  and  standard 
deviation  using  the  relationships: 

N 

=  (Ss)  ■  f  E 

i  =  1 


^i  = 


a.j  +  4m^  +  b^ 
6 
N 

■t  E 

i  =  1 


b^  -  a. 
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where : 


E  (Sj)  *  the  expected  system  size  In  number  of  source  statements 
N  =  the  number  of  system  functions 
E^  =  the  expected  value  of  the  size  of  the  ith  function 

a^,b^  =  the  range  of  the  size  of  the  1th  function  from  smallest 
to  largest 

m^  =  the  most  likely  value  of  the  size  of  the  1th  function 

ar  =  the  estimated  standard  deviation  of  the  system  size 
^s 

=  the  estimated  standard  deviation  of  the  size  or  the 
ith  function. 

The  values  of  the  system  expected  size  and  its  standard  deviation  are 
used  in  the  same  way  as  the  first  method  described  above. 

Level 

This  parameter  is  related  to  the  Difficulty  Graoient, 
discussed  above.  The  Difficulty  Gradient  was  observed  by  Putnin:  zo  assume 
discrete  values  that  are  representative  of  the  type  of  development  asso¬ 
ciated  with  the  system  being  estimated.  Consequently,  Level  measures, 

"  .  .  .  the  amount  of  interfacing,  new  design,  and  concurrent  programming 
that  will  go  on  during  development".*  Level  takes  on  integer  values  from 
1  to  5  depending  on  the  following  considerations: 

(1)  The  system  is  entirely  new  -  designed  and  coded  from  scratch. 

It  has  many  Interfaces  and  must  interact  with  other  systems 
within  a  total  management  Information  system  structure. 

(2)  This  is  a  new  stand-alone  system.  It  is  also  designed  and  coded 
from  scratch  but  is  simpler  because  the  interface  problem  with 
other  systems  is  eliminated. 

*  SLIM  User's  Guide,  p  4-12. 

A-75 


(3)  This  is  a  rebuilt  system  where  large  segments  of  existing  logic 
exist.  The  primary  tasks  are  recoding,  integration,  inter¬ 
facing,  and  minor  enhancements. 

(4)  This  is  a  composite  system  made  up  of  a  set  of  independent 
subsystems  with  few  interactions  and  interfaces  among  them. 
Development  of  the  independent  subsystems  will  occur  with 
considerable  overlap. 

(5)  This  is  a  composite  system  made  up  of  a  set  of  independent 
subsystems  with  a  minimum  of  interactions  and  interfaces  among 
them.  Development  of  the  Independent  subsystems  will  occur 
virtually  in  parallel . 

Past  data  have  shown  that  large  systems  (>200,000  lines)  are  typically  of 
Type  3,  4,  or  5. 

Technology  Factor 

The  Technology  Factor  is  an  integer  parameter  related  to  the  Tech¬ 
nology  Constant  described  above.  SLIM  accepts  values  in  the  range  from 
0  to  22.  The  0  value,  however,  is  simply  a  code  that  indicates  that  the 
model  is  to  provide  the  Technology  Factor.  For  large  systems  the  Techno¬ 
logy  Constant  can  be  expressed  as; 


where : 

C|^  =  The  Technology  Constant 

2.49  =  A  constant  that  represents  the  ratio  of  total  life  cycle 
effort  to  the  design  and  coding  effort  -  it  represents 
the  overhead  labor  associated  with  code  production.  It 
is  valid  for  large  systems,  but  becomes  a  variable  for 
systems  in  the  range  of  18000  to  70000  statements. 
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6  =  A  constant  that  indicates  that  1/6  the  life  cycle  effort  is 

expended  on  logic  design  and  code  production.  Valid  for  large 
systems;  variable  for  intermediate  size  systems, 

=  An  empirical  constant  that  was  shown  above  to  relate  coding 
productivity  and  system  difficulty.  It  assumes  discrete 
values. 

For  large  systems,  then,  the  Technology  Constant  can  be  seen  to  account 
for  the  rate  of  code  production  for  a  given  system  Difficulty,  overhead 
labor  and  the  distribution  of  the  life  cycle  effort.  The  Technology  Factor, 
which  is  used  to  index  the  Technology  Constant  and  which  preserves  its 
discrete  property,  is  said  to  be  "...  a  measure  of  the  state  of 
technology  of  the  human-machine  system"*,  environmental  influences 
and  functional  complexity  of  the  system. 

The  interpretation  of  the  Technology  Factor  is  the  same  for  smaller 
systems  as  it  is  described  above  for  large  systems,  but  the  relationship 
shown  is  modified  in  a  way  that  is  proprietary. 

The  Technology  Factor  is  obtained  by  calibrating  the  model  using 
historical  data  that  are  representative  of  the  project  to  be  estimated. 

The  factor  should  be  stable  in  a  given  organization,  but  should  be  expected 
to  change  to  reflect  differences  in: 

•  Computer  access  and  availability 

•  Software  support  tools,  language 

•  Use  of  modern  programming  practices 

•  Type  of  application 

•  Staff  experience 

•  Customer  relationship 

The  SLIM  manual  indicates  that  few  organizations  are  represented  by 
a  Technology  Factor  greater  than  14  or  less  than  5. 


L.  H.  Putnam,  A  General  Empi-'-^^  Solution,  etc.,  op.  :--.,  353. 
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Additional  Inputs 

SLIM  requires  a  number  of  inputs  in  addition  to  the  ones  described 

above. 


MONTH,  YEAR  =  the  .iTonth  and  year  when  detailed  design  of  the  system 
win  start.  MONTH  is  an  integer  between  1  and  12;  YEAR  is  an 
'Integer  between  40  and  90. 

LABOR  RATE  -  the  fully  burdened  average  S/MY  at  the  user's 
organization. 

STDOEV  -  the  uncertainty  associated  with  the  above  LABOR  RATE. 

INFLATION  RATE  -  the  anticipated  inflation  rate  at  project  start 

ONLINE  -  the  proportion  of  development  that  will  occur  in  online, 
interactive  mode. 

DEVELOPMENT  TIME  -  the  proportion  of  the  development  computer  that 
is  dedicated  to  this  development  effort. 

PRODUCTION  TIME  -  the  proportion  of  the  available  capacity  of  the 
development  computer  that  is  used  for  other  production  work. 

HOL  -  the  proportion  of  the  system  that  will  be  coded  in  a  higher 
order  language. 

LANGUAGE  -  the  primary  language  to  be  used;  should  correspond  to 
the  legend  below. 


(1)  APL  (4)  FORTRAN  (7)  ALGOL  (10)  ASSEMBLER 

(2)  PL/I  (5)  BASIC  (8)  JOVIAL  (11)  RPG 

(3)  COBOL  (6)  CMS  (9)  PASCAL-ADA  (12)  OTHER 


UTILIZATION  -  the  proportion  of  the  memory  of  the  target  machine 
that  will  be  utilized  by  the  end  system. 

real  TIME  CODE  -  the  proportion  of  code  which  is  devoted  to  real 
time  or  time  critical  functions. 
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MODERN  PROGRAMMING  PRACTICES  -  the  4  variables  Include:  STRUCTURED 
PROGRAMMING,  DESIGN/CODE  INSP,  TOP-DOWN  DEVELOPMENT,  and  Chief 
Programmer  Team  usage.  The  responses  for  each  of  these  variables 
should  correspond  to  the  legend  below. 

(1)  <25%  (2)  25-/5%  (3)  >75% 

TYPE  -  description  of  the  type  of  software  system: 

(1)  Real  time  or  time  critical  system 

(2)  Operating  system 

(3)  Command  &  control 

(4)  Business  application 

(5)  Telecommunication  &  message  switching 

(6)  Scientific  system 

(7)  Process  control 

PERSONNEL  EXPERIENCE  -  the  4  variables  Include:  OVERALL,  SYSTEM 
TYPE,  LANGUAGE,  and  HARDWARE.  The  responses  are  used  to  get  an 
Indication  of  the  level  of  personnel  experience  --  overall,  on  a 
system  of  similar  size  and  application,  with  the  programming 
language  to  be  used  on  this  effort,  and  on  the  development 
machine. 

(1)  MINIMAL  (2)  AVERAGE  (3)  EXTENSIVE 

The  Technology  Factor  describes  the  development  environment.  As  was  described 
above,  the  Technology  Factor  is  an  indicator  of  the  efficiency  with  which 
effort  (and  therefore  cost)  is  expended  to  obtain  the  desired  software  system. 
Values  of  the  Technology  Factor  can  be  obtained  from  data  describing  completed 
projects  by  using  the  SLIM  calibration  option.  Given  the.  values  of; 


f  size 

•  Development  Effort 

•  Development  Time 

for  one  or  more  projects,  the  model  calculates  the  Technology  Factors 
that  would  have  produced  the  indicated  experience.  The  user  Is  cautioned 
to  examine  any  Technology  Factor  outside  the  "reasonable  range."  This 
is  Indicated  by  an  asterisk  in  the  printout. 

The  Technology  Factor  may  vary  for  selected  projects  taken  from 
a  single  organization.  This  may  be  caused  by  differences  among  the 
projects  in  computer  access,  software  support,  management  methods  staff 
experience,  language,  user  characteristics,  requirements  stability  and 
functional  complexity.  It  is  the  user's  responsibility  to  examine  these 
possibilities  and  to  verify  that  the  project  being  estimated  is  compatible 
with  the  selected  value. 

Reference 

SLIM  User's  Guide,  Quantitative  Software  Management,  Inc. 
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TECOLOTE 

Description  of  the  Model 

The  Tecolote  provisional  software  cost  estimating  model  was  developed 
to  predict  cost  and  resources  needed  to  develop  tactical  software. 
Specifically  it  was  derived  using  data  representing  Navy  fire  control 
systems  designed  to  operate  against  air  and  sea  threats.  These  two 
classes  of  fire  control  systems  present  different  software  requirements 
because  of  the  effects  of  threat  speed  on  system  response  speed. 

The  justification  for  the  model  form  is  the  hypothesis  that  develop¬ 
ment  effort  is  determined  by  software  size,  system  time  criticality  and 
system  fast  storage  capacity.  The  model  is  limited  to  tactical  systems 
characterized  by  time  criticality,  that  is,  where  the  time  required  to 
access  the  fast  storage  memory  is  comparable  to  the  speed  with  which  the 
computer  is  capable  of  moving  data  during  processing. 

The  software  resource-driving  factors  (storage  capacity  and  time 
criticality)  were  assumed  to  be  predictable  in  terms  of  the  related  threat 
characteristics.  Data  from  five  Navy  software  developments  were  used 
to  relate  the  storage  and  time  requirements  to  threat  size  for  two  speed 
regions  representing  the  air  and  sea  threats. 

A  relationship  was  derived  between  "delivered  code"  (the  total 
code  developed  including  drivers  and  simulators)  and  "operational  code" 

(the  COQO  that  ends  up  in  the  operational  computer).  All  code  size 
measurements  are  in  terms  of  machine  instructions.  The  number  of  machine 
instruct!  is  taken  to  be  the  same  as  the  number  of  words  of  computer 
storage  required  to  store  the  program. 

Operating  instructions  are  related  to  total  fast  storage  capacity 
using  one  sea  threat  data  point  and  two  air  threat  points.  A  relation¬ 
ship  between  total  delivered  instructions  and  operational  instructions 
is  obtained  from  two  data  points. 
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Man-months  of  direct  labor  are  shown  as  functions  of  first  total 
operating  instructions  and  then  total  delivered  instructions. 

A  matrix  of  the  above  relationships  was  prepared  (Table  A-7).  Its  pur¬ 
pose  is  .  .  for  evaluating  software  proposals  from  the  standpoint  of  soft¬ 
ware  design  as  well  as  software  costs."  It  should  be  noted  that  there 
are  redundancies  in  the  relationships  and  that  the  model  does  not  indicate 
which  relationship  should  be  preferred  in  any  given  estimating  situation. 

Labor  and  computer  costs  were  presented  in  1973  dollars  as  func¬ 
tions  of  direct  labor  man-months.  These  were  obtained  from  the  only 
project  for  which  costs  were  available. 

Reference 

Brad  C.  Frederick,  A  Provisional  Model  for  Estimating  Computer  Program 
Development  Costs,  Tecolote  Research,  Inc.,  TM-7,  Dec.  1974. 
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Wol verton 


Description  of  the  Model 

Estimates  of  routine  size  are  converted  to  costs  using  cost  per 
instruction  values  that  are  functions  of  the  routine  tvpe  and  complexity. 

The  costs  are  fully  burdened  and  when  summed  for  all  the  system  routines 
represent  the  total  system  development  cost.  Development  extends  from  analysis 
and  design  through  operational  demonstration.  A  matrix  of  ratios  is  used 
to  allocate  the  total  cost  to  7  phases  with  each  phase  divided  into  up  to 
25  activities.  This  allocation  is  compared  from  the  standpoints  of  staff, 
schedule,  and  general  credibility. 

The  model,  then,  is  a  combination  of  formal  algorithm  and  judgement. 

It  has  been  used  successfully  at  TRW.  As  described  by  Wol verton,  it  features 
a  data  base  of  historical  data  that  provide  the  necessary  cost  per  instruction 
and  allocation  values.  The  procedure  is  adaptable  to  any  new  environment 
by  creating  a  new  data  set  representing  local  definitions  of  phases  and 
activities  and  burdened  cost  conventions.  In  fact,  Wolverton  cautioiis 
that  the  given  values  of  cost  per  instruction  are  for  illustration  and  users 
should  prepare  their  own  values. 

TRW  has  computerized  the  maintenance  of  the  cost  data  base  and  the 
allocation  process.  Given  the  inputs  of  size  and  complexity,  the  system 
calculates  the  cost  allocations  and  facilitates  any  subsequent  adjustments. 
Since  most  models  are  used  in  a  similar  manner,  even  if  the  procedure  for 
using  the  model  does  not  say  so,  there  should  be  no  compromise  of  the 
model's  performance  i^  the  evaluation  is  based  on  a  single  estimate  of  costs. 
Other  adjustments  that  are  necessary  to  execute  the  model  in  di-Pferent 
environments  will  be  discussed  later. 

The  estimating  procedure  begins  by  identifying  all  the  routines 
comprising  the  system.  Each  routine  size,  category,  and  relative  degree 
of  difficulty  are  estimated  by  knowledgeable  persons. 
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The  categories  that  have  "stood  the  test  of  usage"  at  TRW  are: 

•  Control  routine 

•  Input/Output  routine 

•  Pre  or  Post  algorithm  processor 

•  Algorithm 

•  Data  Management  routine 

•  Time-Critical  processor 

Relative  difficulty  is  indicated  by  six  levels  depending  on  whether 
a  routine  is  Old  or  New  and  then  by  simply:  Easy,  Medium  or  Hard, 

The  cost  per  instruction  for  the  36  different  attributes  (6  soft¬ 
ware  categories  by  6  levels  of  difficulty)  is  given  in  Figure  A-7.  Multi¬ 
plying  the  cost  per  instruction  for  each  routine  by  its  number  of  object 
instructions  and  summing  the  products  for  all  the  routines  yields  the 
estimated  total  development  cost. 

The  development  cost  is  allocated  to  the  following  7  phases  using 
proportions  for  each  phase  that  were  obtained  from  the  historical  data  base. 

A.  Performance  and  Design  Requirements 

B.  Implementation  Concept  and  Test  Plan 

C.  Interface  and  Data  Requirements  Specification 

D.  Detailed  Design  Specification 

E.  Coding  and  Auditing 

F.  System  Validation  Testing 

G.  Certification  and  Acceptance  Demonstration 

Then,  the  cost  for  each  phase  is  divided  into  up  to  25  activities 
{ Fables  A-8  and  A-9) , 

A  matrix  of  computer  hours  by  phase  and  software  type  is  used  to 
estimate  conputer  usage  costs  for  development. 


COST  PER  OBJECT 
INSTRUCTION  ($) 


1%  OP  TOTAL  SAMPLE  EXPERIENCING  THIS  RATE.  OR  LESS) 


Figure  A-7.  Cost  Per  Object  Instruction  Versus 
Relative  Degree  of  Difficulty 
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TABLE  A-8 

ACTIVITIES  AS  A  FUNCTION  OF  SOFTWARE  DEVELOPMENT  PHASE 


««ASI 

ACTIViTV 


NOCMAII  I  mOGtAM  I  MOGRAII  I  MOGGAM  1  AHOGAAM  f  MOCRAII 

MANAGtHCNT  i  MAnAGCMiMT  !  HAHAGflliAT  |  mAUAQWt  \  luilAG<irtAT  |  HAAACillEIT 


MOGAAII 

MAMAGEIICHT 


AffOGAA* 

MANAGEHCNT 


AfQUtICIICAn 

OCriAlTIOM 


•cauiAiiifan 

ALLOCATION 


MNEOAMAACC 
ANO  OCIIGM 
“HCOUtAEMCNn  ■ 
(fANT  I) 


CVINT 

GfNfAATlOA  J 
lATHEACt  1 

COMfAAO  I 
ocfnnnoil  4/E  \ 

TCLtMTRt  f 
OEEtliiriON  u9  I 


MOOUCT  COGfiG 
OfTAlifO  TCCN 
OCSCRimOA 

9ANr  m - 

(WTMOUT  LIfniiGS) 


TCCNAtCAL 

OESCAimO* 

UAOArC 


lomtAtE 
MOILCM 
NEPOnn  (vm 


TAAOi  1 

TAAOf 

nuotcs  ' 

truQict 

INTCRfACI  I  »UIICTiOIIAL 
MQm  HCVTli  OC^taiTlO* 


TAAOt 

srgoics 

TAAOi  I 

fTUOlCS  1 

OATA 

OCriAlTlOM 

ALGOfftTMl  I 

OUIGA 

STAAOAAOS 

o«r* 

AAO 

MIC 

COAVfATlOAi 

OEFlAinOA 

sorriNAAt 

ovfAvm 

•VACLlAIAARYt 

PAOOUCT  AMO  MOOVCT  A«0 

COMElGUKATiOa  COAEIGUGATlOa 
COMTAOL  COATtOL 


DATAIAtf 

COHTROL 


Tir 

mauiMniAn 


TWT 

MOCtOGAU 


OfVtlOmCAT 

TCSTtAG 

nANAiAG 

rrsTCH  TUT 
PUAAlAG 

OfVCLOmEAT 

T«T 

software 
SrSTEA  T»r 

OATAUSf 

COATROL 


ACCirTAACE 
AAO  TQT 

aiaaaing 


ACCIPTAACE 

OfHOAfTNAnOA 


(■rccAAfioA 

TUTIA6 


roi  JB»*0«T 

1  Tin  SUFAOAT 

TfsrSUFPORT 

TWT  SUPWAT 

TABLE  A-9 


COST  MATRIX  DATA,  SHOWING  ALLOCATION  OF  RESOURCES 
AS  A  FUNCTION  OF  ACTIVITY  BY  PHASE 


ACTIVITY 

1 

2 

A 

(8) 

10 

8 

JB 

(19) 

6 

3 

C 

(3) 

8 

3 

PHASE 

D 

114) 

6 

3 

£ 

(23) 

7 

3 

F 

(21) 

5 

3 

G 

(12) 

10 

3 

H 

(0) 

2 

5 

3 

6 

4 

6 

3 

8 

5 

4 

13 

8 

5 

6 

5 

5 

4 

2 

5 

5 

2 

2 

3 

3 

2 

2 

1 

6 

22 

8 

7 

12 

3 

7 

5 

8 

7 

10 

8 

7 

2 

6 

8 

7 

5 

9 

6 

• 

10 

17 

10 

10 

8 

11 

2 

10 

10 

9 

7 

6 

12 

2 

5 

13 

13 

4 

7 

10 

3 

5 

14 

4 

3 

15 

5 

25 

16 

4 

4 

5 

4 

4 

10 

10 

17 

3 

6 

5 

6 

5 

8 

10 

18 

5 

6 

3 

8 

5 

2 

5 

10 

19 

10 

15 

14 

5 

20 

10 

21 

2 

4 

5 

5 

7 

9 

9 

3 

22 

5 

6 

8 

4 

23 

24 

4 

3 

2 

3 

5 

3 

25 

25 

2 

1 

2 

3 

5 

10 
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Outputs 

Development  Cost 

The  given  cost  values  are  in  1972  dollars.  The  value  of  cost  results 
from  applying  "bid  rates"  to  labor  costs  which  accounts  for  fringe  benefits, 
overhead,  administrative  expenses  and  other  indirect  costs.  Documentation 
and  travel  costs  are  added  to  the  labor  costs.  Finally,  estimates  are 
made  of  the  computer  costs.  The  distribution  of  the  costs  by  phases  and 
activities  were  described  above. 

Development  Effort 

Cost  is  not  a  suitable  basis  for  evaluating  the  different  software 
estimating  models  because  of  differences  in  accounting  practices  among 
organizations  and  because  of  inflation.  Therefore,  the  Wolverton  cost 
values  were  converted  to  manmonths  using  an  average  burdened  cost  per 
manmonth  of  $4600.  This  value  was  obtained  from  the  article  describing 
the  TRW  estimating  procedure  and,  therefore,  should  be  representative  of 
the  cost  environment. 

Inputs 

Object  Instructions 

The  model  input  measure  of  size  is  applied  to  programs  or  routines. 
These  are  taken  to  be  functionally  distinct  elements  of  a  system  that  would 
be  developed  independently  then  intergrated  into  the  delivered  system.  It 
is  expected  that  these  would  be  independently  operable  using  test  drivers. 
Such  a  definition  is  consistent  with  industry  usage.  The  reference  document 
is  not  specific  on  this  point.  The  term  "instructions"  is  taken  literally. 
This  means  estimating  the  number  of  instructions  in  the  executable  program 
exclusive  of  any  data  areas.  The  number  of  instructions  may  be  estimated 
by  obtaining  the  words  of  memory  occupied  by  the  executable  code  and  dividing 
by  the  average  words  per  instruction. 
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Software  Categories 

Each  routine  is  characterized  according  to  one  of  the  following 
categories : 

C.  Control  Routine.  Controls  execution  flow  and  is  nontime  critical. 

I.  Input/Output  Routine.  Transfers  data  into  and  out  of  computer. 

P.  Pre-or  Post  Algorithm  Processor.  Manipulates  data  for  subsequent 
processing  or  output. 

A.  Algorithm.  Performs  logical  or  mathematical  operations. 

0.  Data  Management  Routine.  Manages  data  transfer  within  the  computer. 
T.  Time  Critical  Processor.  Highly  optimized  machine-dependent  code. 

Degree  of  Difficulty 

Wolverton  indicates  that  any  numeric  representation  of  complexity  may 
be  used.  The  main  purpose  is  to  distribute  the  cost  per  instruction  values 
over  the  range  of  experience  for  a  given  category  of  software.  He  suggests 
a  simple  designation  of  old  or  new,  depending  on  a  loose  interpretation  of 
the  amount  of  reusable  code,  and  easy  medium  or  hard  compared  with  other 
programs  in  the  same  category. 

Reference 

L.  H.  Putnam,  R.  W.  Wolverton,  Tutori  1 ,  Quantitative  Management: 

Software  Cost  Estimating,  IEEE  Computer  Society,  No.  EHO  129-7,  Nov  1977. 
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APPENDIX  B 


WORK  BREAKDOWN  STRUCTURE 


LEVEL  1. 

1.  DEFINITION 

2.  CODING 

3.  DATA  CONVERSION 

4.  INFORiMAL  TEST  AND  INTEGRATION 

5.  FORMAL  TEST  AND  INTE  RATION 

6.  INSTALLATION 

7.  DEVELOPMENT  FACILITIES 

8.  TRAINING 


9.  MANAGEMENT 


LEVEL  2. 


1.  DEFINITION 

.1  SYSTEM  LEVEL 
.2  SYSTEM  SEGMENT  LEVEL 
.3  CPCI  LEVEL 

2.  CODING 

.1  CPCI  LEVEL 

.2  CPRC  LEVEL 

3.  DATA  CONVERSION 

.1  CPCI  LEVEL 

4.  INFORiiAL  TEST  AND  INTEGRATION 

.1  CPCI  LEVEL 

.2  CPRC  LEVEL 

5.  FORMAL  TEST  AND  INTEGRATION 

.1  SYSTEM  LEVEL 
.2  SYSTEM  SEGMENT  LEVEL 
,3  CPCI  LEVEL 

6.  INSTALLATION 

.1  SYSTEM  LEVEL 

7.  DEVELOPMENT  FACILITIES 

.1  SYSTEM  LEVEL 
.2  SYSTEM  SEGMENT  LEVEL 

8.  TRAINING 

.1  SYSTEM  LEVEL 
.2  CPCI  LEVEL 

9.  MANAGEMENT 

.1  SYSTEM  LEVEL 
.2  SYSTEM  SEGMENT  LEVEL 
.3  CPCI  LEVEL 


LEVEL  3. 


1.  DEFINITION 

.1  SYSTEM  LEVEL 

.1  System  Requirements  Collection  and  Definition 
.2  System  Requirements  Evaluation 
.3  System  Design 
.4  System  Design  Verification 

.5  System  Change  Proposal  Evaluation  and  ECP  Preparation 
.6  System  Requirements  Documentation 
.7  System  Design  Documentation 
.8  System  Definition  Monitoring  and  Direction 

.2  SYSTEM  SEGMENT  LEVEL 

.1  Segment  Requirements  Definition 
.2  Segment  Requirements  Evaluation 
.3  Segment  Design 
.4  Segment  Design  Verification 

.5  Segment  Change  Proposal  Evaluation  and  ECP  Preparation 
.6  Segment  Requirements  Documentation 
.7  Segment  Design  Documentation 
.8  Segment  Definition  Monitoring  and  Direction 

.3  CPCI  LEVEL  DEFINITION 

.1  CPCI  Requirements  Collection  and  Definition 
.2  CPCI  Requirements  Evaluation 
.3  CPCI  Design 
.4  CPCI  Design  Verification 

.5  CPCI  Change  Proposal  Evaluation  and  ECP  Preparation 
.6  CPCI  Requirements  Documentation 
.7  CPCI  Design  Documentation 
.8  CPCI  Definition  Monitoring  and  Direction 
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LEVEL  3  (Con't) 


2.  CODING 

.1  CPCI  LEVEL 

.1  CPCI  Level  Cod’ng 

.2  CPCI  Code  Documentation 

.3  Monitoring  and  Direction  of  CPCI  Coding 


.2  CPRC  LEVEL 

.1  CPRC  Level  Coding,  Compilation,  and  Inforr.al  Review 


3.  DATA  CONVERSION 

.1  CPCI  LEVEL 

.1  CPCI  Data  Conversion 

.2  Data  Conversion  Documentation 

.3  Monitoring  and  Direction  of  CPCI  Data  Conversion 


4,  INFORMAL  TEST  AND  INTEGRATION 
.1  CPCI  LEVEL 

.1  Informal  CPCI  Test,  and  Integration  Planning 
.2  Informal  CPCI  Test  and  Integration  Conduct 
.3  Documentation  of  Informal  Test  and  Integration 
.4  Inforrrial  CPCI  Test  and  Integration  Monitoring  and  Direction 

,2  CPRC  LEVEL 

.1  CPRC  Test  and  Integration  Planning 
.2  CPRC  Test  and  Integration  Conduct 
.3  Documentation  of  CPRC  Test  and  Integration 
,4  CPRC  Test  and  Integration  Monitoring  and  Directives 
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LEVEL  3  (Con't) 


5.  FORMAL  TEST  AND  INTEGRATION 

.1  SYSTEM  LEVEL  DT&E 

.1  System  DT&E  Planning 

.2  System  DTSE  Procedure  Development 

.3  System  DT&E  Execution 

.4  System  DT&E  Data  Reduction 

.5  System  DT&E  Error  Identification 

.6  System  DT&E  Documentation 

.7  System  DT&E  Monitoring  and  Direction 

.2  SYSTEM  SEGMENT  LEVEL  DT&E 

.1  Segment  Test  Planning 

.2  Segment  Test  Procedure  Development 

.3  Segment  Test  Execution 

.4  Segment  Test  Data  Reduction 

.5  Segment  Test  Error  Identification 

.6  Segment  Test  Documentation 

.7  Segment  Test  Monitoring  and  Direction 

.3  CPC  I  LEVEL  DT&E 

.1  CPCI  Qualification  Test  Planning 

.2  CPCI  Qualification  Test  Procedure  Development 

.3  CPCI  Qualification  Test  Execution 

.4  CPCI  Qualification  Test  Data  Reduction 

.5  CPCI  Qualification  Test  Error  Ide,.tification 

.6  CPCI  Qualification  Test  Documentation 

.7  CPCI  Qualification  Test  Monitoring  and  Direction 


6.  INSTALLATION 

.1  SYSTEM  LEVEL 

.1  Planning  for  Installation 
.2  Site-Specific  Adaptation 
.3  Installation  Documentation 
.4  Installation  Monitoring  and  Direction 
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LEVEL  3  (Con't) 


7.  DEVELOPMENT  FACILITIES 
.1  SYSTEM  LEVEL 

.1  Development  Facility  Planning 

.2  Development  Facility  Site  Preparation 

.3  Development  Facility  Equipment  Acquisition 

.4  Development  Facility  Equipment  Maintenance 

.5  Development  Facility  Software  Acquisition 

.6  Development  Facility  Software  Maintenance  and  Modification 

.7  Development  Facility  Operation 

.8  Development  Facility  Documentation 

.9  Monitoring  and  Direction  of  Development  Facility  Provision 
and  Operation 

.2  SYSTEM  SEGMENT  LEVEL 


8.  TRAINING 


.1  SYSTEM  LEVEL 

.1  System  Training  Planning 

.2  System  Training  Material  Development 

,3  Instruction  in  System  Use,  Operation,  and  Maintenance 

.4  System  Training  Documentation 

.5  Monitoring  and  Direction  of  System  Training 


.2  CPCI  LEVEL 

.1  CPCI  Training  Material  Development 
.2  Instruction  in  CPCI  Use,  Operation,  and  Maintenance 
.3  CPCI  Training  Documentation 
.4  Monitoring  and  Direction  of  CPCI  Training 
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LEVEL  3  (Con’t) 


9.  MANAGEMENT 

.1  SYSTEM  LEVEL 

.1  System  Management  Planning 
.2  System  Project  Direction 
.3  System  Configuration  Management 
.4  Reporting  System  Development  Status 


.2  SYSTEM  SEGMENT  LEVEL 

.1  Segment  Managemeit  Planning 
.2  Segment  Development  Direction 
.3  Segment  Configuration  Management 
.4  Reporting  Segment  Development  Status 

.3  CPCI  LEVEL 

.1  CPCI  Management  Planning 
.2  CPCI  Development  Direction 
.3  CPCI  Configuration  Management 
.4  Reporting  CPCI  Development  Status 
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APPENDIX  C 

MODEL  ESTIMATING  PERFORMANCE 


C-1 


TAB_E  C-1 

model  estimating  performance  -  AEROSPACE  CORPORATION,  COMMERICAL 


MODEL;  AEROSPACE  CORPORATION 

DATA  SET;  COMMERCIAL 


CASE 

MM 

ACTUAL 

MM* 

EST 

ACT 

EST 

1 . 

A1 

127.2 

190.9 

0.666 

2. 

A2 

38.0 

130.5 

0.291 

3. 

A3 

48.7 

163.3 

0.298 

4. 

A4 

29.3 

115.2 

0.237 

5. 

A5 

45.5 

189.6 

0.240 

6. 

A6 

44.9 

216.2 

0.208 

7. 

A7 

30.5 

118.2 

0.258 

8. 

A8 

53.0 

169.5 

0.313 

9. 

A9 

232.8 

307.1 

0.758 

10. 

AlO 

211.1, 

274.6 

0.769 

11. 

All 

13.8 

119.2 

0.116 

Mean 

79.5 

0.378 

Standard 

76.2 

0.234 

Deviation 

RMS  ERROR;  107. 

RELATIVE  RMS  ERROR;  1.35 


*  Support  software  relati onstiip  {^ne  Appendix  A) 
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TABLE 


MODEL  ESTIMATING  PERFORMANCE  -  AEROSPACE  CORPORATION,  DSDC 

MODEL:  AEROSPACE  CORPORATION 

DATA  SET:  DSDC 


1 

i 

CASE 

1 

MM 

ACTUAL  i 

MM  * 

EST 

ACT 

EST 

1 . 

DC 

1 

2.3  ; 

79.4 

i 

0.0290 

2. 

DK  j 

79.9 

134.8 

0.583  1 

3. 

DS 

8.8 

101.5 

0.0867 

4. 

DU 

2.9 

95.5  ! 

0.0304 

5- 

FB 

88.9 

169.1 

0.526 

1 

FD 

7.1 

134.8 

0.0527 

7. 

FE 

4.9 

125.2 

0.3091 

1  S. 

FF 

2.6 

103.7 

0.0251 

9. 

BH 

17.8 

144.2 

0.123 

10. 

1 

66 

9.5 

105.4 

0.0901 

11. 

GG 

48.8 

122.3 

0.399 

12. 

BI 

3.5 

76.4 

0.0458 

13. 

ZP 

9.8 

31.0 

0.121 

14. 

US 

172.9 

190.0 

0.910 

15. 

JD 

45.4 

111.4 

0.408 

16. 

QD 

23.0 

153.3 

0.150 

17. 

DO 

247.5 

104.3 

2.37 

Mean 

45.6 

0.553 

Standard 

68.8 

0.579 

Deviation 

RMS  ERKOR:  96.0 

RELATIVE  RMS  ERROR:  2.11 


*  Support  software  relationship  (See  Appendix  A) 


TABLE  C-3 

MODEL  ESTIMATING  PERFORMANCE  -  AEROSPACE  CORPORATION,  SEL 

MODEL:  AEROSPACE  CORPORATION 

DATA  SET:  SEL 


CASE 

MM 

ACTUAL 

FiM* 

EST 

1 . 

AA 

111.0 

209.3 

2. 

AB 

221.3 

182.7 

3. 

AC 

254.3 

209.3 

4. 

AD 

268.8 

187.9 

5. 

AE 

324.4 

239.9  1 

6. 

AF 

77.6 

160.5  1 

7. 

AG 

53.1 

119.5 

8. 

AH 

29.0 

117.8 

9. 

AI 

79.8 

157.0 

10. 

AT 

19.5 

84.7 

n. 

AK 

25.3 

79.0 

12. 

AL 

20.5 

103.0 

13. 

AM 

13.9 

60.7 

14. 

AN 

23.5 

8.31 

15. 

AO 

276.8 

218.5 

16. 

AP 

67,0 

119.1  1 

17. 

AQ 

43.8 

104.7 

Mean 

112.  i 

Standard 

Deviation 

109.  j 

ACT 

EST 


0.530 
1.21 
1.22 
1 .43 
1 .35 
0.484 
0.44., 
0.246 
0.508 
0.230 
0.320 
0.199 
0.229 
0.283 
1.27 
0.563 
0.418 
0.643 
0.450 


RMS  ERROR: 

RELATIVE  RMS  ERROR; 


67.8 

0.605 


★ 


Support  software  re1atiooshi"i  (See  Apoendix  4  5 


TABLE  C-4 

MODEL  ESTIMATING  PERFORMANCE  -  BOEING  COMPUTER  SERVICES,  DSDC 


MODEL:  BOEING  COMPUTER  SERVICES 

DATA  SET:  DSDC 


CASE 

MM 

ACTUAL 

MM 

EST 

wmM 

1. 

DC 

2.4 

4.4 

0.542 

2. 

DK 

82.7 

15.0 

5.52 

3. 

DS 

9.1 

7.6 

1 .19 

4. 

DU 

3.0 

6.5 

0.465 

5. 

FB 

92.0 

30.8 

2.98  j 

6. 

FD 

7.4 

13.5 

0.550 

7. 

FE 

5.1 

16.0 

0.319 

8. 

FF 

2.7 

7.8 

0.345  I 

9. 

BH 

18.4 

24.1 

0.764 

10. 

BB 

9.8 

'  9.8 

0.997 

n. 

GG 

50.5 

14.8 

3.42  1 

12. 

BI 

3.6 

3.7 

0.983 

13. 

ZP 

10.1 

4.7 

2.14  j 

14. 

US 

178.9 

161.2 

1.11  1 

15. 

JD 

47.0 

10.5 

4.48  1 

16. 

Mean 

OD 

23.8 

34.2 

31.4 

0.757  ' 

1.66  1 

Standard 

Deviation 

48.1 

1.60  1 

1 

_ L 

RMS  ERROR:  26.9 

RELATIVE  RMS  ERROR:  0.787 


TABLE  C-5 

MODEL  ESTIMATING  PERFORMANCE  -  OOD  MICRO  PROCEDURE,  DSCC 

MODEL:  DOD  MICRO  ESTIMATING  PROCEDURE 

DATA  SET:  DSDC 


CASE 

MM 

ACTUAL 

— 

MM 

EST 

ACT 

EST 

1. 

DC 

2.4 

2.7 

0.882 

2. 

OK 

82.7 

14.0 

5.90 

3, 

DS 

9,1 

4.4 

2.06 

4. 

DU 

3.0 

7.7 

0.391 

5 . 

FB 

92.0 

26.0 

3.5s 

6. 

FD 

7  .4 

2.2 

3.29 

7. 

FE 

5.1 

99.6 

0.0512 

8. 

FF 

2.7 

6.5 

0.414 

9. 

BH 

18,4 

74.5 

0.247 

10, 

BB 

9.8 

9.3 

1.05 

11 . 

GG 

50.5 

9.6 

5.24 

12. 

BI 

3.6 

1  .9 

1  .89 

13. 

ZP 

10.1 

6.8 

1  .49 

14. 

US 

178.9 

121 .7 

1  .47 

15. 

JD 

47.0 

6.0 

7.89 

16. 

QD 

23.8 

68.6 

0.347 

t  Mean 

34,2 

i 

2.26  i 

Standard 

Deviation 

48.1 

2.32  1 

1 

_ 

RMS  ERROR:  43.2 

REU\TIVE  RMS  ERROR:  1.26 


C-6 


TABLE  C-6 

MODEL  ESTIMATING  PERFORMANCE  -  DOTY  ASSOCIATES,  INC,  DSDC 

MODEL:  DOTY  ASSOCIATES,  INC. 

DATA  SET:  DSDC 


CASE 

MM 

ACTUAL 

MM  * 

EST 

■■ 

ACT 

EST 

1. 

DC 

1.0 

7.1 

0.141 

2. 

DK 

32.6 

23.2 

1.41 

3. 

DS 

4.9 

12.3 

0.398 

4. 

DU 

1.1 

10.7 

0.103 

5. 

FB 

38.8 

38.5 

1 .01 

6. 

FD 

3.9 

23.2 

6.00 

7. 

FE 

2.4 

19.6 

0.122 

8. 

FF 

1.7 

12.9 

0.131 

9. 

BH 

6.0 

26.9 

0.223 

10. 

BB 

2.8 

13.4 

0.209 

11. 

GG 

8.9 

18.6 

0.479 

12. 

BI 

1.9 

6.5 

0.292 

13. 

ZP 

5.7 

7.4 

0.770 

14. 

US 

48.3 

50.0 

0.966 

15. 

JD 

21.6 

15.1 

1.43 

16. 

QD 

0.8 

0.0259 

Mean 

11.4 

i 

0.725 

Standard 

Deviation 

15.3 

0.966 

RMS  ERROR: 


12.0 


relative  RMS  ERROR:  1.05 


*  Business  programs  relationship  (See  Appendix  A) 


C-7 


TABLE  C-7 

MODEL  ESTIMATING  PERFORMANCE  -  FARR  AND  ZAGORSKI,  DSDC 

MODEL:  FARR  AND  LAGORSKI 

DATA  SET:  OSOC 


CASE 

MM 

ACTUAL 

MU  * 

EST 

- 1 

ACT  1 

EST  1 

1. 

DC 

1.0 

100. 

I 

1 

1 

0.0100  j 

2. 

DK 

32.6 

230. 

0.142  j 

'  J  . 

OS 

4.9 

147. 

0.0333  1 

j 

ru 

1.1 

131 . 

0.00840  j 

5. 

FB 

38.8 

343. 

0.113  j 

6. 

FO 

3.9 

00 

o 

0.0188  1 

7. 

FE 

2.4 

163. 

0.0147  j 

8. 

FF 

1.7 

140. 

0.0121  j 

9. 

BH 

6.0 

266. 

0.0226 

10. 

SB 

2.8 

155. 

0.0181 

11. 

GG 

8.9 

192. 

0.0464 

12. 

BI 

1.9 

94. 

0.0202 

13. 

ZP 

5.7 

106. 

0.0538 

U, 

US 

48.3 

464. 

0.104 

15. 

JO 

21.6 

178. 

0.121 

16. 

qo 

0.8 

285. 

0.00281 

17. 

OJ 

21.2 

147. 

0.144 

Mean 

12.0 

0.0521  1 

Standard 

Deviation 

15.0 

1 

0.05C9  1 

1 

RMS  ERROR:  203. 

RELATIVE  RMS  ERROR:  16.9 


*  Relationship  (3)  (See  Appendix  A) 

C-8 


TABLE  C-8 

MODEL  ESTIMATING  PERFORMANCE  -  PRICE  S,  COMMERCIAL 


MODEL:  PRICE  S 

DATA  SET:  COMMERCIAL 


CASE 

MM 

ACTUAL 

MM 

EST 

ACT 

1.  A1 

87.2 

48 

1.82 

2.  A2 

22.9 

17 

1.35 

3.  A3 

38.8 

44 

0.882 

4.  A6 

37.2 

71 

0.524 

5.  A8 

40.7 

12 

3.39 

6.  A9 

192.0 

176 

1.09 

7.  An 

10.5 

17 

0.618 

Mean 

61.3 

1.38 

Standard 

Deviation 

62*.  3 

0.990 

RMS  ERROR:  23.5 

RELATIVE  RMS  ERROR:  0.383 


C-9 


TABLE  C-9 

MODEL  ESTIMATING  PERFORMANCE  -  PRICE  S,  DSDC 
MODEL:  PRICE  S 

DATA  SET:  DSDC 


CASE 

MM 

ACTUAL 

MM 

EST 

ACT 

nr 

1 . 

DC 

1  .7 

36 

.0472 

2. 

DU 

2.4 

8 

.300 

3. 

FD 

5.5 

11 

.500 

4. 

FE 

3.8 

14 

.271 

5. 

FF 

2.0 

9 

.222 

6. 

BH 

14.5 

19 

.763 

7. 

BB 

7.4 

10 

.740 

8. 

GG 

40.2 

12 

3.35 

9. 

ZP 

8.0 

6 

1.33 

10. 

US 

129.5 

35 

3.70 

11. 

JO 

35.2 

11 

3.20 

12. 

QD 

11.7 

22 

.532 

Mean 

21. S 

36.2 

1.35 

RMS 

ERROR: 

31.4 

- 

RELATIVE  RMS  ERROR:  1.44 


C-10 


TABLE  C-10 

MODEL  ESTIMATING  PERFORMANCE  -  PRICE  S,  SEL 


i>10DEL:  PRICE  S 

DATA  SET:  SEL 


CASE 

f'lfl 

ACTUAL 

MM 

EST 

ACT 

EST 

1. 

AA 

39.6 

48 

0.825 

2. 

AB 

79.0 

85 

0.929 

3. 

AC 

90.7 

96 

0.945 

4. 

AD 

95.9 

78 

1.23 

5. 

AE 

115.7 

150 

0.771 

6. 

AF 

27.7 

27 

1.03 

7. 

AG 

18.9 

22 

0.860 

8. 

AH 

10.3 

22 

0.470 

9. 

AI 

28.5 

39 

0.730 

10. 

AJ 

7.0 

5 

1.39 

n. 

AK 

9.0 

10 

0.902 

12. 

AL 

7.3 

16 

0.458 

13. 

AN 

8.4 

n 

0.764 

Mean 

41.1 

0.870 

Standard 

Deviation 

39.4 

0.260 

RMS  ERROR:  12.3 

RELATIVE  RMS  ERROR:  0.297 


c-n 


table  c-n 

MODEL  ESTIMATING  PERFORMANCE  -  SLIM,  COMMERCIAL 


MODEL;  SLIM 

DATA  SET;  COMMERCIAL 


CASE 

MM 

ACTUAL 

MM 

EST 

ACT 

m 

1.  A1 

38.5 

1.84 

2.  A3 

38 

20.7 

1.84 

3.  AS 

32 

37.1 

0.863 

4.  A6 

36 

34.0 

1.06 

5.  A9 

184 

212.1 

0.868 

6.  AlO 

163 

138.5 

1.18 

Mean 

87.3 

1.27 

Standard 

Deviation 

68.5 

0.454 

RMS  ERROR;  21.5 

RELATIVE  RMS  ERROR;  0.246 


C-12 

i 


I 


UBLE  C-12 

MODEL  ESTIMATING  PERFORMANCE  -  SLIM,  OSOC 


MODEL:  SLIM 

DATA  SET:  DSDC 


CASE 

Mfl 

ACTUAL 

MM 

EST 

ACT 

i?r 

1.  FB 

81.3 

79.8 

n 

2.  BH 

17.4 

27.1 

3.  US 

155.6 

125.4 

■g 

Mean 

84.8 

Standard 

Deviation 

69.2 

m 

RMS  ERROR:  18.3 

RELATIVE  RMS  ERROR:  0.216 


C-13 


TABLE  C-13  V 

MODEL  ESTIMATING  PERFORMANCE  -  SLIM.  SEL 
MODEL:  SLIM 

DATA  SET:  SEL 


I  CASE 

i _ 

Mii 

.  ACTUAL 

1 - 

MM 

j  EST 

L-  - . 

ACT 

EST 

1. 

AA 

39.6 

1  . 

45.1 

0.878 

2. 

AS 

79.0 

76.9 

1 .03 

3. 

AC 

90.7 

90.0 

1 .01 

4. 

AD 

95.9 

102.7 

0.934 

5. 

AE 

115.7 

307.9 

0.376 

6. 

AI 

28.5 

44.6 

0.639 

7. 

A6 

138.3 

148.2 

.  0.933 

8. 

A7 

98.4 

179.7 

0.548 

Mean 

85.8 

Standard 

Deviation 

36.6 

RMS  ERROR:  74.2 

RELATIVE  RMS  ERROR:  0.865 


C-14 


TABLE  C-14 

MODEL  ESTIMATING  PERFORMANCE  -  TECOLOTE,  DSDC 

MODEL;  TECOLOTE 

DATA  SET:  DSDC 


CASE 

MM 

ACTUAL 

ACT 

EST 

1. 

- 1 

DC 

2.4 

38.1 

0.0629 

2. 

DK 

82.7 

1  193.1 

0.428 

3. 

DS 

9.1 

30.8 

0.113 

4. 

DU 

3.0 

67.1 

0.0447 

5. 

FB 

92.0 

387.4 

0.238 

6. 

FO 

7.4 

193.3 

0.0383 

7. 

FE 

5.1 

153.9 

0.0331 

8. 

FF 

2.7 

86.4 

0.0313 

9. 

BH 

18.4 

23^7 

0.0774 

10. 

BB 

9.8 

90.7 

0.108 

n. 

GG 

50.5 

143.2 

0.353 

12. 

BI 

3.6 

33.8 

0.106 

13. 

ZP 

10.1 

40.5 

0.250 

14. 

US 

1  178.9 

554.5 

0.323 

15. 

JO 

47.0 

107.6  ' 

0.437 

16. 

QD 

!  23.8 

236.7 

0.0830 

Mean 

34.2 

0.170 

Standard 

48.1 

1 

0 . 580 

Deviation 

i 

1 

i 

RMS  ERROR:  168. 


RELATIVE  RMS  ERROR:  4.92 

*  Estimating  relationship  using  number  of 
operating  instructions. 


C-15 


TABLE  C-05 

MODEL  ESTIMATING  PERFORMANCE  -  WOLVERTON,  DSDC 


MODEL:  WOLVERTON 

DATA  SET:  DSDC 


CASE 

MM 

ACTUAL 

MM 

EST 

Kl 

1 . 

DC 

2.4 

3.4 

0.712 

2. 

DK 

82.7 

45.4 

1 .82 

3. 

OS 

9-1 

6.4 

1  1.43 

4. 

DU 

3.0 

24.8 

I  0.121 

5. 

FB 

1 

92.0 

124.7 

0.738 

6. 

FD 

7.4 

16.8 

0.441 

7. 

FE 

5.1 

45.5 

0.112 

8. 

FF  1 

2.7 

29.7 

0.0910 

9. 

BH 

18.4 

11.6 

1.59 

10. 

66 

9.8 

23.2 

0.422 

n. 

GG  1 

50,5 

23.6 

2.14 

12. 

BI 

3.6 

8.1 

0.445 

13. 

ZP 

10.1 

6.9 

1 .46 

14. 

US  1 

178.9 

106.5 

1 .68 

15. 

JO 

47.0 

17.7 

2.65 

16. 

qo  1 

23.8 

48.6 

0.49 

Mean 

1 

i 

34.2 

1.02 

Standard 

Deviation 

48.1 

i 

i _ 

0.802 

RMS  ERROR;  31.7 


RELATIVE  RMS  ERROR: 


0.927 


TABLE  C-1<5  ■ 

MODEL  ESTIMATING  PERFORMANCE  -  RECALIBRATED  SIZE  EQUATION 


C- 


TABLE  C-17 

SUMMARY  OF  MODEL  ESTIMATING  PERFORMANCE 
RMS  ERROR* 


DATA 


SET 


MODEL  TYPE 


REGRESSION 
r  AEROSPACE 
B  DOTY 

C  FARR  S  ZAGORSKI 
D  TECOLOTE 
E  (al^) 


HE"RISTIC 
F  BOEING 
G  COD  MICRO 
H  PRICE  S 
I  WOLVERTON 


PHENOMENOLOGICAL 
J  SLIM 


CO^,MERCIAL  0  S  D  C  S  E  L 


APPENDIX  D 

DESCRIPTION  OF  MODEL  INPUTS 


TABLE  D-1 

AEROSPACE  AND  TECOLOTE  MODELS 


SUMMARY  OF  INPUTS 


COMMERCIAL 

DSDC 

SEL  j 

OBJ. 

OBJ. 

SYS. 

SYS. 

INS. 

SYS. 

INS. 

L 

A1 

78335 

DC 

8943 

AA 

98402 

2. 

A2 

30543 

DK 

33090 

AB 

70306 

3. 

A3 

53200 

OS 

16390 

AC 

98402 

4. 

A4 

22459 

DU 

14110 

AD 

75357 

5. 

A5 

77039 

FB 

58010 

AE 

137944 

6. 

A6 

106573 

FD 

33120 

AF 

5103C 

7. 

A7 

23939 

FE 

27560 

AG 

24575 

8. 

A8 

58391 

FF 

17290 

AH 

23726 

9- 

A9 

254232 

BH 

39120 

AI 

48273 

10. 

A10 

192706 

B6 

* 

17990 

AJ 

10476 

11. 

All 

24420 

GG 

26000 

AK 

8823 

12. 

BI 

8116 

AL 

17022 

13. 

ZP 

9383 

AM 

4596 

14. 

US 

77470 

AN 

9985 

15. 

JD 

20640 

AO 

109466 

16. 

QD 

45510 

AP 

24392 

17. 

DJ 

17560 

AQ 

_ 

17725 
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SUMMARY  OF  INPUTS 


TABLE  D-5 

FARR  &  ZAGORSKI  MODEL 
SUMMARY  OF  INPUTS 

DSDC  DATA 


1 


TABLE  D-6 
PRICE  S 

SUMMARY  OF  INPUTS 


COMMERCIAL 


TABLE 
PRICE 
SUMMARY  OF 

OSD' 


D-7 


S 

INPUTS 


1.00 

r 

1.00 

0.90 

1.00 

4000 

1.00 

1.00 

0.90 

l.-O 

4000 

1.00 

1.00 

0.20 

1.00 

4000 

1.00 

1 .00 

1 .00 

1.00 

4000 

1.00 

1.00 

1 .00 

4000 

1.00 

1.00 

1 .00 

1 .00 

4000 

1.00 

1 .00 

1 .00 

4000 

1.00 

1.00 

1.00 

1 

4000 

1.00 

1.00 

1.00 

4000 

1.00 

■  . 

1.00  i 

, 

1.00  ' 

4000 

1.00 

1.00 

1 

1.00  1 
i 

4000 

1.00 

1.00 

_ 

1 .00 

1.00 

_ 

4000 

TABLE  D-8 


PRICE  S 

SUMMARY  OF  INPUTS 
SEL 


SUMMARY  OF  INPUTS 


SUHMARY  OF  INPUTS 


K 


TABLE  D-11 

RECALIBRATED  SIZE  EQUATION 
SUMMARY  OF  INPUTS 


COMMERCIAL 

DSOC 

SEL 

SYSTEM 

STMTS 

SYSTEM 

STMTS 

SYSTEM 

STMTS 

1 

1. 

A) 

28879 

DC 

2205 

AA 

13400 

2. 

9605 

DK 

10182 

AB 

25000 

3. 

18373 

_DS 

4479 

AC 

26800 

4. 

A4 

6706 

DW 

3761 

■  AD 

23750 

5. 

A5 

28321 

FB 

19624 

AE 

52350 

6. 

A6 

41384 

FO 

10190 

AF 

17500 

7. 

A7 

1046 

FE 

8222 

AG 

6900 

8. 

A8 

5950 

FF 

4768 

AH 

5300 

9. 

A9 

114325 

BH 

12381 

AI 

13250 

TO. 

AlO 

82687 

BB 

4993 

AJ 

2200 

n. 

All 

7395 

GO 

7680 

AK 

2100 

12. 

BI 

1970 

AL 

4700 

13. 

ZP 

23^4 

AM 

1050 

14. 

US 

27519 

AN 

2450 

15. 

JO 

5865 

AO 

41300 

16. 

QD 

14776 

AP 

6606 

17. 
_ 1 

AQ 

5077  ; 

D-12 


references 


1.  Major  System  Acquisitions,  Dept,  of  Defense  Directive  5000,1, 

Jan.  18,  1977. 

2.  Major  System  Acquisition  Process,  Dept,  of  Defense  Directive  5000.2, 

Jan.  18,  1977. 

3.  Economic  Analysis  and  Program  Evaluation  for  Resource  Management, 

Air  Force  Regulation  1978-1,  Dec.  23,  1973. 

4.  Computer  Technology  Forecast  and  Weapon  Systems  Impact  Study  (COMTEC-2000) , 
Vol.  II,  Technical  Data,  NTIS,  AD  B034955L,  Dec  1978. 

5.  A.  Ash,  D.  1/,  Kelliher,  J,  P.  Locker,  III,  T,  Connors,  DoD  Weapon  Systems 
Software  Acquisition  and  Management  Study,  Vol.  I,  MITRE  Findings  and 
Recommendations,  MITRE  Corporation,  MTR-6908,  May  1975. 

6.  T.  G.  James,  Jr.,  Software  Cost  Estimating  Methodology,  NAECON  77 
Record  pp  22-28. 

7.  David  A.  Fisher,  Automatic  Data  Processing  Cost  in  the  Defense  Department, 
Institute  for  Defense  Analyses,  Paper  P-1046,  Oct  1974. 

8.  Barry  W.  Boehm,  Software  and  Its  Impact:  A  Quantitative  Assessment, 
Datamation,  May  1974,  pp  48-59. 

9.  E.  N.  Dodson,  E.  E.  Salkovich,  and  W.  E.  Waller,  Advanced  Cost  Estimating 
and  Synthesis  Techniques  for  Avionics,  General  Research  Corporation 
CR-2-461,  Sept  1975. 

10.  E.  N.  Dodson,  et.al..  Cost  Estimating  Relationships  for  Airborne  Radars 
FLIRS,  and  Avionics  Logistics  Support,  General  Research  Corporation, 
CR-1-559. 

11.  M.  Eddins  Earles,  Factors,  Formulas  and  Structures  for  Life  Cycle 
Costing,  Eddins-Earles,  Concord,  Mass.,  1978. 

12.  P.  F.  Ostwald,  Cost  Estimating  for  Engineering  and  Management, 
Prentice-Hall,  Englewood  Cliffs,  New  Jersey,  1974. 

13.  E.  N.  Dodson,  Studies  of  Improved  Techniques  for  Parametric  Cost 
Analysis,  General  Research  Corporation,  IM-1957,  Dec  1974. 

14.  E.  H.  Yates,  Interrelationships  of  Technology,  System  Performance, 
and  Prices  for  Mini/Midi  Computers,  General  Research  Corporation, 

TIO  2286,  Aug  1980. 

15.  J.  R.  Brown,  Impact  of  MPP  on  System  Development,  Rome  Air  Development 
Center,  RADC-TR-77-121 . 


R1 


16. 

17. 


18. 

19. 

20. 

21. 


22. 


23. 


24. 


25. 


26. 

27. 

28. 

29. 

30. 


Summary  Notes  of  a  Government/Industry  Software  Sizing  and  Costing 
Workshop,  National  Technical  Information  Service,  AD-A026964,  Oct  1974. 

W.  S.  Junk,  A  Software  Cost  Estimation  Methodology  Creating  the 
Structure  for  Reliable  Application  of  State  of  the  Art  Cost  Models, 
Workshop  in  Quantitative  Software  Models,  IEEE  Cat.  No.  TH0067-9, 

Oct  1979,  pp  56-62. 

R.  L.  Smith,  Estimating  Software  Project  Resource  Requirements, 

Vol .  aI,  Structured  Programming  Series,  Rome  Air  Development  Center, 
RADC-TR-74-300,  Vol.  XI,  Jan  1975. 


G.  L.  Meyers,  Estimating  the  Costs  of  a  Programming  System  Development 
Project,  IBM,  Systems  Development  Division,  IBM  TR00.2316,  May  1972. 


Ouantitative  Software  Models,  Data  and  Analysis  Center  for  Software, 
RADC,  SRR-1 ,  Mar  1979. 


T.  J.  Devenny,  An  Exploratory  Study  of  Cost  Estimating  at  the  E-ectronics 
Systems  Division,  NTIS,  AD  A030162. 

L.  H.  Putnam,  R.  W.  Wolverton,  Quantitative  Management  Software  Cost 
Estimating,  IEEE  Cat.  No.  EHO  129-7. 

W.  S.  Junk,  J.  A.  McCall,  L.  H.  Putnam,  G.  F.  Walters,  Survey  of 
Software  Cost  Estimating  Techniques,  General  Electric,  Information 
Systems  Programs,  78CIS010,  May  1978. 


J.  A.  Clapp,  A  Review  of  Software  Cost  Estimation  Methods,  NTIS  AD  A029748, 
Aug  1976. 

M.  Finfer,  R.  Mish,  Software  Acquisition  Management  Guidebook:  Cost 
Estimation  and  Measurement,  NTIS,  AD  A055574,  Mar  1978. 


M.  L.  Shooman,  Tutorial  on  Software  Cost  Models,  Workshop  on  Quantitative 
Software  Models,  IEEE  Cat.  No,  TH0067-9,  Oct  1979,  pp  1-19. 

G.  H.  Sandler,  B.  I.  Rachowitz,  Software  Cost  Models  -  Grumman  Experience, 
Workshop  on  Quantitative  Software  Models,  IEEE  Cat.  No.  TH0067-9,  pp  69-77. 

T.  G.  James,  Jr.,  D.  V.  Ferens,  Application  of  the  RCA  PRICE-S  Software 
Cost  Estimation  ftodel  to  Air  Force  Avionics  Laboratory  Programs,  AFSC, 

AF  Avionics  Laboratory,  AFAL-TR-79-n64,  Oct  1979. 

R.  E.  Steffey,  Jr.,  An  Analysis  of  the  RCA  PRICE-S  Cost  Estimation  Model 
as  it  Relates  to  Current  Air  Force  Computer  Software  Acquisition  and 
Management,  Thesis,  AFIT,  GSM/SM/79D-20,  Dec  1979. 

J.  Schneider,  IV,  A  Preliminary  Calibration  of  tne  RCA  PRICE  S  Software 
Cost  Estimation  Model,  Thesis,  NTIS,  AO  A046808,  Sept  1977. 


R-2 


31.  J.  I'/.  Bailey,  V.  R.  Basil i,  A  Meta-Model  for  Software  Development 
Resource  Expenditures,  Dept,  of  Computer  Science,  dniv.  of  Maryland. 

32.  B.  Curtis,  Measurement  and  Experimentation  in  Software  Engineering, 
Proceedings  of  the  IEEE,  Vol .  68,  No.  9,  Sept  1980,  p  1151. 

33.  V.  R.  Basili,  R.  W.  Reiter,  Jr.,  An  Investigation  of  Human  Factors 
in  Software  Development,  Computer.  Dec  1979,  pp  21-38. 

34.  J.  B.  Glore,  Software  Acquisition  Management  Guidebook:  Life  Cycle 
Events,  AFSC,  Electronic  Systems  Division,  ESD-TR-77-22,  Feb  1977. 

35.  D.  R.  Peterson,  Software  Acquisition  Management  Guidebook:  Software 
Development  and  Maintenance  Facilities,  NTIS,  AD  038234,  Apr  1977. 

36.  P.  V.  Norden,  Useful  Tools  for  Project  Management,  Management  of 
Production,  M.  K.  Starr,  Ed.,  Penguin  Books,  1970,  pp  71-101 . 

37.  P.  V.  Norden,  Project  Life  Cycle  Modelling:  Background  and  Application 
of  the  Life  Cycle  Curves,  Software  Life  Cycle  Management  Workshop, 

US  Army  Computer  Systems  Command,  Aug  1977,  pp  217-306. 

38.  L.  H.  Putman,  A  General  Empirical  Solution  to  the  Macro  Software  Sizing 
and  Estimating  Problem,  IEEE  Transactions  on  Software  Engineering, 

Vol.  SE-4,  No.  4,  July  1978,  pp  345-361. 

39.  M.  H.  Halstead,  Elements  of  Software  Science,  Elsevier  North-Hol land,  1977. 

40.  L.  A.  Belady,  M.  M.  Lehman,  The  Evolution  Dynamics  of  Large  Programs, 

IBM  Watson  Research  Center,  Sept  1975. 

41.  R.  Thibodeau,  E.  N.  Dodson,  Life  Cycle  Phase  Interrelationships, 

Journal  of  Systems  and  Software,  Vol  1,  No.  3,  1980,  pp  203-211. 

42.  S.  S.  Yau,  J.  S.  Collofello,  Performance  Ripple  Effect  Analysis  for 
Large  Scale  Software  Maintenance,  AFSC,  Rome  Air  Development  Center, 
RADC-TR-80-55,  Mar  1980. 

43.  C.  A.  Graver,  E.  E.  Balkovich,  W.  M.  Carriere,  R.  Thibodeau,  Cost 
Reporting  Elements  and  Activity  Cost  Tradeoffs  for  Defense  Systems 
Software,  General  Research  Corporation,  CR-721 ,  Mar  1977. 

44.  W.  M.  Carriere,  R.  Thibodeau,  Development  of  a  Logistics  Software  Cost 
Estimating  Technique  for  Foreign  Military  Sales,  General  Reserach 
Corporation,  CR-3-839,  Jun  1979. 

45.  T.  Yamane,  Statistics,  An  Introductory  Analysis,  Harper  &  Row,  2nd  Ed, 

1967,  pp  70?lTrT: 

v 

46.  F.  P.  Brooks,  The  Mythical  Man-Month,  Datamation,  Dec  1974,  pp  45-52. 


R-3 


1 


47.  Reference  Manual  PRICE  Software  Model,  RCA/PRICE  Systems. 

48.  SLIM  User’s  Guide,  Quantitative  Management  Systems,  Inc. 

49.  J.  H.  Herd,  J.  N.  Postak,  W.  E.  Russell,  K,  R.  Stewart,  Software 

Cost  Estimating  Study,  Study  Results,  Vol .  I,  AFSC  Rome  Air  Development 
Center,  RADC-TR-77-220,  Jun  1977. 

50.  E.  A.  Nelson,  Management  Handbook  for  the  Estimation  of  Computer 
Programming  Costs,  System  Develofmient  Corporation,  RM-3225/000/01 , 

Mar  1967. 

51.  R.  Thibodeau,  The  State-of-the-Art  in  Software  Error  Data  Collection 
and  Analysis,  NTIS  AD  A075228,  Jan  1978. 

52.  R.  Thibodeau,  The  Feasibility  of  Obtaining  Software  Research  Data  at 
the  US  Army  Computer  Systems  Command,  Army  Institute  for  Research  in 
Management  Information  and  Computer  Science  (AIRMICS),  Jul  1980. 

53.  S.  Pelosi,  B.  Parham,  J.  Berterman,  Arlen  Feldman,  Automatic  Data 
Processing  Resource  Estimating  Procedures  (ADPREP),  NTIS,  AO  711117, 

Aug  1970,  p  23. 

54.  R.  W.  Wolverton,  The  Cost  of  Developing  Large-Scale  Software,  IEEE 
Transactions  on  Computers,  Jun  1974,  p  626. 

55.  C.  E.  Walston,  Working  Group  on  Software  Cost,  Workshop  Quantitative 
Software  Models,  IEEE  Cat.  No.  TH0067-9,  Oct  1979,  p  241. 


R-4 


MISSION 

of 

Rome  Air  Devebpment  Center 

RAVC  pianA  and  executes  fie^taKch,  dzvtlopmznt,  ttit  and 
Adeemed  acqiuj>ltA.on  paagaam  in  iuppofit  oi  Command,  Control 
Ccmmanicatiom  and  Int^iUganct  [Ch]  actioiticA.  Technical 
and  engineating  ^uppoat  mtkin  aaeoi  oi{  technical  competence 
ii  provided  to  ES'O  Paagaam  0<^icei  IPO4)  and  othea  ESP 
elemental.  The  paincipal  technical  mi^iion  aaeai  ane 
communication,  electxomagnetic  guidance  and  control,  6ux- 
oeillance  of,  ground  and  aexoipace  objects,  intelligence  data 
collection  and  handling,  Infonmatlon  igitem  technology, 
ionoiphexic  propagation,  6olld  itate  6ciencet ,  microwave 
phyiici  and  electronic  reliability,  maintainability  and 
compatibility . 


